genhtml: --keep-going doesn't work for ERROR: <file> is not readable or doesn't exist.

[ntnx-aleksa]

Hello, we're facing an error handling issue in genhtml after we switched to lcov 2.0.

genhtml version

$ genhtml --version
genhtml: LCOV version 2.0-1

Command:

genhtml --demangle-cpp --keep-going --rc genhtml_hi_limit=80 --rc genhtml_med_limit=40 ./coverage_merged.info
...
genhtml: ERROR: <path> is not readable or doesn't exist. See the '--substitute' and '--synthesize-missing' genhtml options for methods to fix the missing path or ignore the problem.

Why isn't this error ignorable using --ignore-errors source or --keep-going? The man page clearly states:

source: The source code file for a data set could not be found.

This seems like a breaking change compared to lcov 1.x.

[henry2cox]

If this was an ignorable error, what you want the tool to do when it can't find the source or the source is missing, and you also told it that you wanted it to generate the source code view?

What it does do is to tell you that it can synthesize a "source" view for you, if you want. Or you can exclude the file in question, or you can tell it where to find the actual code.

There are a couple of ways to tell the tool how to find your code. Often the simplest is to --substitute away the issue. More complicated scenarios are typically easier to solve with a --resolve-script. Similarly, if you have specified a --annotate-script, then the tool will get the code (and annotate data) from your revision control system; the source file itself does not need to exist locally.

[ntnx-aleksa]

I'm aware of the possible solutions, however I disagree that this is not an ignorable error. The previous lcov versions simply did not include the source file in the HTML output and discarded the annotated counters. This is the behavior that should be present by default in genhtml, and --synthesize-missing, --substitute and --resolve-script should all be alternatives to this. Not to mention that this directly contradicts the semantics of --keep-going:

Do not stop if error occurs: attempt to generate a result, however flawed.

[henry2cox]

Sounds like you have a solution: use the prior version.

The doc used to say some errors are not ignorable - but it seems to no longer do so. That's an issue that I will (eventually) fix.

[ntnx-aleksa]

I would like to describe a reason for considering to implement this behavior in lcov 2.x that I hope might make sense. We've decided to use --synthesize-missing to resolve the case of missing source files. The source files whose paths cannot be resolved are not of interest and we would like to not have them in the HTML report. If we have them synthesized, they come as a nuisance to the report and influence the final numbers. It's also not obvious in a directory listing that the source file is synthesized. In this case, it's really useful for genhtml to filter these out for us by ignoring this specific error. Filtering them explicitly with --exclude isn't possible, because they can appear in various places and it's hard to write a glob pattern for that. I'm not sure whether --resolve-script could be used to decide to skip a source file, if so, that would be helpful for this case. Thank you for considering to resolve this problem.

[henry2cox]

Ah. That makes sense. Not 100% safe, but reasonable. Hope somebody is there to review the messages. I'm curious the source of those missing files. (Locally, they are often generated code where the directives are a bit messed up. There are not a huge number - so we either exclude or synthesize them.)

I added --filter missing which will skip files which are not locally visible or if a --resolve-script callback is defined and the callback returns "" (empty string) or undef. The latter is to handle the case that the source is not local but is available in your revision control system Filters report a count at the end of execution - so you should be able to tell if something goes off the rails and a lot of unexpected stuff got excluded.

[ntnx-aleksa]

I now better understand your concerts about the previous version's behavior - it doesn't make for a more accurate report. I took some time to investigate the source of the missing files, and it seems to be a Python C wrapper whose artifact is an .so file that's referencing .c files that are actually temporary artifacts created from existing .py files. This is on Clang 13, gcov-compatible code coverage.

Thank you for considering to add the filter, it's useful for us in these situations where we need a report, even though it might not be the most accurate.

[henry2cox]

I believe this issue is fixed - and will go ahead and close it a few days from now if there are no further updates/nothing that needs to be done.

[henry2cox]

Closing this issue now. If there is still a problem, please feel free to reopen this issue or file a new one. Please describe the problem in detail and provide a testcase which illustrates the issue, if possible.