Code Coverage is wrong when using test based defines

[IBeRyUS]

Hi all, We are having an issue with code coverage when we use test based preprocessor definitions. If we enable test based preprocessor macros ceedling doesn't combine test coverage data.

Also created a repository for testing about the issue. https://github.com/IBeRyUS/ceedling_gcov_test Below you can find %100 code coverage yml file without using test preprocessors(they are commented out).

---
:project:
  :use_exceptions: FALSE
  :use_test_preprocessor: TRUE
  :use_auxiliary_dependencies: TRUE
  :build_root: build
  :test_file_prefix: test_
  :which_ceedling: gem
  :ceedling_version: 0.30.0
  :default_tasks:
    - test:all

:environment:
:extension:
  :executable: .out

:paths:
  :test:
    - -:test/support
    - test/**
  :source:
    - src/**
  :support:
    - test/support
  :libraries: []

:defines:
  :common: &common_defines
    - TEST
  :test:
    - *common_defines
#  :test_sum:
#    - *common_defines
#    - SUM

:cmock:
  :mock_prefix: mock_
  :when_no_prototypes: :warn
  :enforce_strict_ordering: TRUE
  :plugins:
    - :ignore
    - :callback
  :treat_as:
    uint8:    HEX8
    uint16:   HEX16
    uint32:   UINT32
    int8:     INT8
    bool:     UINT8

:gcov:
  :reports:
    - HtmlDetailed
  :gcovr:
    :html_medium_threshold: 75
    :html_high_threshold: 90

:libraries:
  :placement: :end
  :flag: "-l${1}"
  :path_flag: "-L ${1}"
  :system: []    # for example, you might list 'm' to grab the math library
  :test: []
  :release: []

:plugins:
  :load_paths:
    - "#{Ceedling.load_path}"
  :enabled:
    - stdout_pretty_tests_report
    - module_generator
    - gcov
...

Output of this unit test is :

Test results without test specific preprocessor macro

Test 'test_div.c' ----------------- Compiling test_div.c... Compiling calculations.c with coverage... Linking test_div.out... Running test_div.out... Test 'test_mult.c' ------------------ Linking test_mult.out... Running test_mult.out... Test 'test_sub.c' ----------------- Linking test_sub.out... Running test_sub.out... Test 'test_sum.c' ----------------- Compiling test_sum.c... Linking test_sum.out... Running test_sum.out... -------------------------- GCOV: OVERALL TEST SUMMARY -------------------------- TESTED: 4 PASSED: 4 FAILED: 0 IGNORED: 0 --------------------------- GCOV: CODE COVERAGE SUMMARY --------------------------- calculations.c Lines executed:100.00% of 17 calculations.c Branches executed:100.00% of 2 calculations.c Taken at least once:50.00% of 2 calculations.c No calls Could not find coverage results for src/main.c

After enabling test specific preprocessor macros test like below code coverage is become wrong.

  :test_sum:
    - *common_defines
    - SUM

Test results with test specific preprocessor macro

Test 'test_div.c' ----------------- Compiling unity.c... Compiling cmock.c... Linking test_div.out... Running test_div.out... Test 'test_mult.c' ------------------ Generating runner for test_mult.c... Compiling test_mult_runner.c... Compiling test_mult.c... Linking test_mult.out... Running test_mult.out... Test 'test_sub.c' ----------------- Generating runner for test_sub.c... Compiling test_sub_runner.c... Compiling test_sub.c... Linking test_sub.out... Running test_sub.out... Test 'test_sum.c' ----------------- Updating test definitions for test_sum Generating runner for test_sum.c... Compiling test_sum_runner.c... Compiling test_sum.c... Compiling calculations.c with coverage... Linking test_sum.out... Running test_sum.out... Restored defines and build path to standard -------------------------- GCOV: OVERALL TEST SUMMARY -------------------------- TESTED: 4 PASSED: 4 FAILED: 0 IGNORED: 0 --------------------------- GCOV: CODE COVERAGE SUMMARY --------------------------- calculations.c Lines executed:22.22% of 18 calculations.c Branches executed:0.00% of 2 calculations.c Taken at least once:0.00% of 2 calculations.c No calls Could not find coverage results for src/main.c

Using test specific macros makes ceedling to create new environment for specific tests and it prevents code coverage to show properly.

Do you know is there any way for to combine them?

I already tried use_deep_dependencies and didn't work out. I find out using gcov and lcov manually will allow combining these tests but we have in-house CI system and using manual commands won't work for server.

Thanks, Ibrahim

[CezaryGapinski]

Hi @IBeRyUS !

It's not currently supported in the latest ceedling release. I was working on this feature some time ago. Actually I'm waiting for the new gcovr release where some bugs related to unwrapping filepaths on Windows will be fixed. I'm going to create Pull Request after that.

If you are really interested here are changes on my branch with new :merge_coverages: option is available. You can try it if you are using Linux. I will be happy if someone else also test this new feature.

[IBeRyUS]

Hi @CezaryGapinski Thank you very much for your response and sorry about late reply. Yes we are working on Linux and I will test it. I will pull your branch and add as local ceedling than test it. This is a background task for moving from another unit test environment to ceedling and going little slow because it has not high priority enough. Will respond output here.

[NFerragut]

Hi @CezaryGapinski, I have the same problem, but I'm using ReportGenerator instead of gcovr. You mentioned that you are waiting for the new gcovr release. Are you proposing a gcovr-specific solution?

[CezaryGapinski]

Hi @NFerragut At this moment I don't have any universal solution for both ReportGenerator and gcovr I would try to use gcovr because is able to create JSON reports which can next be merged into one final report. I've seen that ReportGenerator has some feature to merge multiple reports into one but I'm not sure how does it works.

[pez3]

Hello

i'm currently facing the same problem. spent quite some time to finally get here and to find out, that this is not supported for now...

since this is a quite old information... is there now a solution for this problem?

just for information: when running the tests with gcov i get the following message

libgcov profiling error: ... .gcda:overwriting an existing profile data with a different timestamp

best regards peter

[SilverBulletEngineering]

@pez3 If you provide your yml file and more details including logs and errors and a detail about what aspect isn't working I bet Mark will be able to provide more help - or another ceedlinger. Thanks.

[pez3]

hi,

of course i can provide more information! i have a driver file where i can select via define HTR_CUR_MEAS_EN whether this driver shall use current or voltage measurement algorithm. this define i'm setting different via the yml file.

when i build and run the unit tests for this file, i see that first test_heater_with_current_measurement is built and run. afterwards test_heater_with_voltage_measurement is built as desired. then, wenn running the tests the following error appears in console:

libgcov profiling error:/Test/Unit/build/gcov/out/heater.gcda:overwriting an existing profile data with a different timestamp

i also see in the coverage report, that the part with current measurement is not covered.

so i think the tests for the heater source file are correctly built with different defines, but the coverage report is not merged but overwritten.

my project yml looks like this:

unittest.txt

thank you for your help! greets peter

[SilverBulletEngineering]

Check out stack overflow and see if there are useful posts or post your own. https://stackoverflow.com/questions/39502855/gcov-profilling-error-error-writing

https://stackoverflow.com/questions/68136230/gcda-file-coverage-info-not-getting-updated-when-two-executables-run-for-same-s/68149742#68149742