Incremental coverage for an incremental build

[abussy-aldebaran]

Hi,

I saw issue #157 but I'm not sure if it's what I'm looking for.

I would like to do incremental coverage for the initial capture of a C++ incremental build. For example, I build two cpp files and generate the base coverage (the lcov command is incorrect, but I hope the intent is clear)

a.cpp  ->  a.gcno
b.cpp  ->  b.gcno
lcov --initial --capture "a.gcno b.gcno" -o coverage_base.info

Now I erase the .gcno, modify b.cpp and rebuild. gcc only rebuilds b.cpp, as expected :

b.cpp  ->  b.gcno
lcov --initial --capture "b.gcno" -o coverage_base_increment.info

Is there a way to merge coverage_base_increment.info with coverage_base.info, where all data from b.cpp in coverage_base.info is overwritten by the data from coverage_base_increment.info? Is that already the behavior of --add-tracefile ?

lcov -a coverage_base.info -a coverage_base_increment.info  -o coverage_base_merged_info

[henry2cox]

I am not sure that I understand your question or your intent. However, I think that you are trying to simply generate a tracefile from your code, after you made a few changes. You had generated a tracefile with an earlier version of code.

If you simply try to merge the previous and current data (via lcov -a), then lcov will try to combine the data from your new version of b.cpp with the data from your old version of b.cpp. Since the file has been edited, that is unlikely to be what you want. (Also note that if you use the lcov --version-script ... feature, then the tool will notice that the file version has changed and will refuse to simply merge the incompatible data.)

Another observation is that all the coverpoints will hae a zero hit count when using lcov --initial ... - so I'm really not sure why you would want or need to merge as simply recapturing from the set of current files will give you what I think you wanted.

I think that a better approach is for you to describe what you are trying to accomplish in your development environment (i.e., your high level goal). There may be some more appropriate way to do that.

[abussy-aldebaran]

My team's C++ project takes a long time to build. In our CI, we try to avoid rebuilding everything from scratch, and we keep the build tree between consecutive builds. So .gcno files are also kept. A new build will (in theory) rebuild only the parts of the code that changed, thus updating only the corresponding .gcno. Processing all .gcno with lcov --initial --capture takes a long time, so I would like to speed it up using the fact that only a few of them changed since the previous processing.

[henry2cox]

if you have a reasonable server, you could try lcov --parallel ... That ought to go quite a bit faster (we see near linear in server core count, up to about 22 or so - then flattening improvement). You can also add the --profile flag - to collect some performance statistics, so we can see where time is going, to see if something can be improved. Note that, while the lcov/2.0 release introduced the --parallel flag, there was an enhancement a couple of weeks ago which had a substantial effect on capture performance.

Can you quantify "[lcov] takes a long time"? How long? Approximately how many files? (At least one user has reported capture from ~42K files in ~15 minutes. We see a few minutes, for the LLVM code base...though I can't currently remember exactly how many minutes.)

[abussy-aldebaran]

I'll do some more testing to get the correct stats

[henry2cox]

BtW: unlikely to be faster, and a lot more complicated (hence, easy to make a mistake) - but another way you could do what I think you want is:

1. find all the .gcno files that got updated, moved, or removed from the previous build Use lcov --capture -o remove_me.info ... to turn them into a .info file (of all the stuff you want to ignore/remove). There are several ways to do this - e.g., via --include or --exclude
2. Use set operations to remove them from the previous data: `lcov -o orig_remains.info orig.info --subtract remove_me.info
3. Now merge this previous remainder with your new data from changed files: lcov -o new_merged.info -a orig_remains.info -a new_updates.info

It should be the case that the result you get is the same as you would have gotten by capturing everything from the current build. (diff or a differential coverage report should be ale to check this.)

This is definitely the long way around the block. (And not what we use the intersect and difference operations for, internally.)

The other thing I forgot to ask is what you are using a --initial capture for. Why not just capture the result (after running regressions)?

[abussy-aldebaran]

For the processing of 1k3 .gcno, I got

• > 20min with lcov 1.16
• ~9min with lcov on master with -j 1 This is very far from 42k in 15min. Nothing seems to stand out in the profile though.

[henry2cox]

Can you try again with -j 0 (which tells the tool to count the number of cores on your server, then use that number of slaves). The result ought to be near linear in the number of cores. I'm pretty sure the large example was run on a quite substantial server - I did not run the experiment, only got the report.

-j 1 says "use 1 thread/one core - regardless of how many might be available". Interesting (and actually not expected) that TOT lcov appears to be about 2X faster than 1.16, in single core mode.

[abussy-aldebaran]

The excessive time taken by lcov 1.1x seems to be caused by something in our CI, but I haven't been able to figure what yet.

In an other environment I got :

• 6m30s with lcov 1.1x
• 9m45s with lcov on master and -j 1
• 3min with lcov on master and -j 0

Is this more in line with what's expected?

In any case, as this is not a problem from lcov, I'm closing the issue. Thanks for the help!

[henry2cox]

Is this more in line with what's expected?

I'd actually expect it to be quite a bit faster than that, with -j 0 (unless you have a 2 core machine - in which case, this is exactly what I would expect) - depending on the size of your example (number of compilation units) and possibly some other factors. Running lcov --profile prof.json -j 0 .... will write a bunch of runtime information to file 'prof.json'. That may (or may not) help to identify any lingering issues. You can either send that to me (via github, or by deducing my actual email), or can take a look yourself. There is a spreadsheet.py application in the scripts directory of the release which turns the data into a more readable form.

[abussy-aldebaran]

Sorry, I omitted the precision, my machine has 8 cores.

I figured where my problem came from. I'm cross-compiling and the toolchain, that includes the C++ STL, is sadly in the same tree structure as lcov's input directory, in the CI only. So they were not considered as externals. I was filtering with --remove after the captures. Excluding them right from the start with --exclude fixed the problem.