LCOV 2.1 not compatible with Codecov

[albinahlback]

At FLINT we have used LCOV along with Codecov to get code coverage. Yesterday I tried to switch to LCOV 2.1, but that did not work on Codecov's end. Not sure who's to blame, I just know that switching back to LCOV 2.0 works. Here are the commits, changing it back to LCOV 2.0: https://github.com/flintlib/flint/commit/13e69eb2fe4768fecb2a9896e92cee23b727298c and https://github.com/flintlib/flint/commit/b63929ba29df710a3415d2067611b00affd6ab02.

Any guess to why this is?

While still on the topic: Do you know why --omit-lines in LCOV 2.0 seems to work with Codecov? We want to avoid including aborting functions, and so we push --omit-lines "flint_throw", but that doesn't seem to have an effect in Codecov. I think it worked first, but doesn't seem to work now.

[henry2cox]

Without some more context, I fear that I have no idea :-( lcov/2.1 is more picky than 2.0, and also checks some additional things; this could be tripping you up.

Do you have a log which shows errors and/or warnings? Can you download v2.1 and then execute the capture and/or genhtml steps in your sandbox - to see what is happening?

Unrelated - but I noticed in your changelog that you are omitting lines which contain only close brace. lcov --filter brace ... will do something similar - but will leave braces which it thinks may be significant (e.g., implicit 'else' destinations). While I definitely agree that this is a hack - it is a slightly less dangerous hack than blanket removal.

[albinahlback]

Thanks for your quick response!

Unrelated - but I noticed in your changelog that you are omitting lines which contain only close brace. lcov --filter brace ... will do something similar - but will leave braces which it thinks may be significant (e.g., implicit 'else' destinations). While I definitely agree that this is a hack - it is a slightly less dangerous hack than blanket removal.

I think we only had a problem with functions ending with an if-else statement, hence the regex ^}$ (I think it was mainly void type functions). But thanks for the tip!

Do you have a log which shows errors and/or warnings? Can you download v2.1 and then execute the capture and/or genhtml steps in your sandbox - to see what is happening?

Here is the Codecov error (yes, not a very good error code). Here is the associated runner. Based on the runner log, it looks exactly the same as with 2.0, so it appears as if it works.

But I will download both versions locally and try to identify the difference.

[henry2cox]

I fear that I don't know what I am looking at :-( - but when I check the 'Ubuntu gcc code coverage x10" page in the above link - it appears that geninfo was able to capture the coverage data (though, with quite a few 'inconsistent' warnings). Which build failed? Or is this job after reverting back to lcov/2.0?

[albinahlback]

This was the one with 2.1. I am currently running all tests locally to see what the difference is on my system.

[albinahlback]

I ran both LCOV 2.0 and LCOV 2.1 on my system, and I pasted the output from them below. I took the releases here on Github.

Command-line output from LCOV 2.0

``` $ lcov -j $(expr $(nproc) + 1) --capture --rc geninfo_unexecuted_blocks=1 --ignore-errors unused --omit-lines "flint_throw" --omit-lines '^}$' --exclude "/usr/*" --exclude "src/*/inlines.c" --exclude "src/test/main.c" --exclude "src/test/t-*.c" --exclude "src/*/test/main.c" --exclude "src/*/test/t-*.c" --exclude "src/generic_files/exception.c" --exclude "src/generic_files/profiler.c" --exclude "src/profiler.h" --directory build --output-file build/lcov_2.0/coverage.info Capturing coverage data from build geninfo cmd: '/usr/local/bin/geninfo build --output-filename build/lcov_2.0/coverage.info --ignore-errors unused --rc geninfo_unexecuted_blocks=1 --exclude /usr/* --exclude src/*/inlines.c --exclude src/test/main.c --exclude src/test/t-*.c --exclude src/*/test/main.c --exclude src/*/test/t-*.c --exclude src/generic_files/exception.c --exclude src/generic_files/profiler.c --exclude src/profiler.h --omit-lines flint_throw --omit-lines ^}$ --parallel 5 --memory 0' Found gcov version: 13.2.1 Using intermediate gcov format Writing temporary data to /tmp/geninfo_datl5_N Scanning build for .gcda files ... Found 5121 data files in build files remaining: 4921 (61.77 files/s - predict 1.33 minutes remaining) files remaining: 4721 (54.94 files/s - predict 1.43 minutes remaining) files remaining: 4521 (49.44 files/s - predict 1.52 minutes remaining) files remaining: 4321 (44.30 files/s - predict 1.63 minutes remaining) files remaining: 4121 (36.97 files/s - predict 1.86 minutes remaining) files remaining: 3921 (33.41 files/s - predict 1.96 minutes remaining) files remaining: 3721 (28.77 files/s - predict 2.16 minutes remaining) files remaining: 3521 (26.61 files/s - predict 2.21 minutes remaining) files remaining: 3321 (23.51 files/s - predict 2.35 minutes remaining) files remaining: 3121 (22.07 files/s - predict 2.36 minutes remaining) files remaining: 2921 (19.62 files/s - predict 2.48 minutes remaining) files remaining: 2721 (18.91 files/s - predict 2.40 minutes remaining) files remaining: 2521 (18.24 files/s - predict 2.30 minutes remaining) files remaining: 2321 (16.80 files/s - predict 2.30 minutes remaining) files remaining: 2121 (16.22 files/s - predict 2.18 minutes remaining) files remaining: 1921 (15.71 files/s - predict 2.04 minutes remaining) files remaining: 1721 (14.75 files/s - predict 1.95 minutes remaining) files remaining: 1521 (13.90 files/s - predict 1.82 minutes remaining) files remaining: 1321 (13.10 files/s - predict 1.68 minutes remaining) files remaining: 1121 (12.82 files/s - predict 1.46 minutes remaining) files remaining: 921 (12.54 files/s - predict 1.22 minutes remaining) files remaining: 721 (12.29 files/s - predict 0.98 minutes remaining) files remaining: 521 (12.01 files/s - predict 0.72 minutes remaining) files remaining: 321 (11.72 files/s - predict 0.46 minutes remaining) files remaining: 121 (11.46 files/s - predict 0.18 minutes remaining) Excluded data for 3326 files due to include/exclude options Finished .info-file creation Omitted 18013 total lines matching 2 '--omit-lines' patterns geninfo: WARNING: ('unused') 'exclude' pattern 'src/generic_files/exception.c' is unused. geninfo: WARNING: ('unused') 'exclude' pattern 'src/generic_files/profiler.c' is unused. (use "geninfo --ignore-errors unused,unused ..." to suppress this warning) ```

Command-line output from LCOV 2.1

``` $ lcov -j $(expr $(nproc) + 1) --capture --rc geninfo_unexecuted_blocks=1 --ignore-errors unused --omit-lines "flint_throw" --omit-lines '^}$' --exclude "/usr/*" --exclude "src/*/inlines.c" --exclude "src/test/main.c" --exclude "src/test/t-*.c" --exclude "src/*/test/main.c" --exclude "src/*/test/t-*.c" --exclude "src/generic_files/exception.c" --exclude "src/generic_files/profiler.c" --exclude "src/profiler.h" --directory build --output-file build/lcov_2.1/coverage.info Capturing coverage data from build geninfo cmd: '/usr/local/bin/geninfo build --output-filename build/lcov_2.1/coverage.info --ignore-errors unused --rc geninfo_unexecuted_blocks=1 --exclude /usr/* --exclude src/*/inlines.c --exclude src/test/main.c --exclude src/test/t-*.c --exclude src/*/test/main.c --exclude src/*/test/t-*.c --exclude src/generic_files/exception.c --exclude src/generic_files/profiler.c --exclude src/profiler.h --omit-lines flint_throw --omit-lines ^}$ --parallel 5' Found gcov version: 13.2.1 Using intermediate gcov format geninfo: WARNING: (usage) branch filter enabled but branch coverage not enabled (use "geninfo --ignore-errors usage,usage ..." to suppress this warning) Recording 'internal' directories: /home/albin/cpkg/flint/build Writing temporary data to /tmp/geninfo_datXM5e Scanning build for .gcda files ... Found 5121 data files in build using: chunkSize: 819, nchunks:7, intervalLength:256 elapsed:0.4m: remaining:4302 files 2.3m: 31.27 files/s 0.03 s/file (interval:31.27 f/s 0.03 s/f) elapsed:0.4m: remaining:3483 files 0.9m: 62.16 files/s 0.03 s/file (interval:5216.93 f/s 0.03 s/f) elapsed:0.4m: remaining:2664 files 0.5m: 92.54 files/s 0.03 s/file (interval:4082.04 f/s 0.03 s/f) elapsed:0.4m: remaining:1845 files 0.3m: 122.31 files/s 0.03 s/file (interval:3503.05 f/s 0.03 s/f) elapsed:0.5m: remaining:1026 files 0.1m: 147.52 files/s 0.03 s/file (interval:840.45 f/s 0.03 s/f) elapsed:0.7m: remaining:0 files 0.0m: 124.43 files/s 0.03 s/file (interval:76.59 f/s 0.02 s/f) Finished processing 5121 GCDA files Apply filtering.. Filter: chunkSize 100 nChunks 52 Finished filter file processing Finished .info-file creation Summary coverage rate: source files: 5191 lines.......: 87.4% (266609 of 305178 lines) functions...: 79.7% (13941 of 17498 functions) Omitted 36026 total lines matching 2 '--omit-lines' patterns Message summary: 1 warning message: usage: 1 ```

However, by a simple diff I can see that there is no difference between the outputs, which should mean that the error is either on my part (most likely, I suppose) or Codecov.

I will look into it some more, then perhaps close it.

[henry2cox]

Looks like there is a counting bug in either lcov/2.0 or lcov/2.1 as well: the number of omitted line matches is exactly doubled. Looks suspicious (though maybe not hugely important).

I don't think there are any format differences between the data generated by the two version - but you might want to diff those as well (maybe you already have).

[albinahlback]

Looks like there is a counting bug in either lcov/2.0 or lcov/2.1 as well: the number of omitted line matches is exactly doubled. Looks suspicious (though maybe not hugely important).

Yes, I noticed it as well. I did run LCOV directly one after the other -- does this affect the result in any way?

I don't think there are any format differences between the data generated by the two version - but you might want to diff those as well (maybe you already have).

I'm a bit of a novice when it comes to this. I checked the command-line output as well as the file related to --output-filename. What other type of files should I check?

[henry2cox]

Yes, I noticed it as well. I did run LCOV directly one after the other -- does this affect the result in any way?

No - should not make any difference.

If you diffed the stdout/stderr and also your coverage.info file - then that's it. Nothing else to look at (unless you are interested to really get into the weeds of temporary/intermediate files). I'm not familiar with codecov - but I suspect that it is only interested in the .info file - maybe also the stdout and stderr logs as well, if it is checking for unexpected messages/unexpected behaviour.
It almost certainly does not care about details of processing order, record order in the output files, etc.

[albinahlback]

Thanks for your all of your inputs. It looks like it is on Codecov's end in this case.