genhtml - file is not readable or doesn't exist error

[chris-wickens]

I've been seeing this issue on one of my Codemagic Mac M1 machines intermittently:

genhtml: WARNING: ('source') "/Users/builder/clone/packages/SF:/Users/builder/clone/apps/project_mobile/lib/app/am_app.dart" does not exist: No such file or directory (see --synthesize-missing option to work around) genhtml: ERROR: /Users/builder/clone/packages/SF:/Users/builder/clone/apps/project_mobile/lib/app/am_app.dart is not readable or doesn't exist. See the '--substitute' and '--synthesize-missing' genhtml options for methods to fix the missing path or ignore the problem.

The file does exist, and running the same build again led to the error magically disappearing. Running the command locally on my M1 macbook is worse - it fails every time, picking a random file to complain is missing.

genhtml coverage/lcov.info -o coverage/html

Reading tracefile coverage/lcov.info. Found 70 entries. Found common filename prefix "/Users/chriswickens/code/project-mobile-alt" Generating output. Processing file some_random_file.dart lines=22 hit=0 is not readable or doesn't exist. See the '--substitute' and '--synthesize-missing' genhtml options for methods to fix the missing path or ignore the problem.

It's not clear to me what could be causing this. Running genhtml with the --debug flag indicates the lcov.info file is corrupt, but ignoring the package and corrupt errors gets me back to the same issue, and lcov viewer is happy to load it. I'm seeing this issue on the latest release and the master branch. Any help would be greatly appreciated!

[henry2cox]

Do you have a testcase that reproduces the issue (even intermittently?)

I had three other questions:

• if you are using --parallel - please remove it. Parallelism will introduce changes in processing order which might be related to what you see - and might explain why the issue comes and goes. (I don't know of any such bugs, but I would not be hugely surprised if there was one. Parallel processing is important - and so would be a high priority issue)
• does your file path really include "SF:" in the middle? It ought not to cause a problem - but SF: is the delimiter that the lcov .info file format uses for file names. It is conceivable that there is a bug that is picking up the wrong part of your file name. (This seems unlikely as that bug would not be intermittent.)
• does your result really emit a warning and then an error with no newline? (That ought not to be happening either...not that that would explain an intermittent error either.)

[chris-wickens]

Thanks for the quick response. No test case as of yet, I'll look at putting one together.

• I'm not specifying parallel, but I'm not sure if it's disabled by default.
• Good spot, no that's likely being added by the script I used to merge multiple lcov files together. I forgot I switched to a different lcov.info generator for my local builds which doesn't add the SF: in the middle, but that's giving me the same error message. It does mean that the error is probably valid for my CI build though, and the intermittency might be an issue with just my local machine.
• I checked the CI log and the warning and error are on separate lines, so that's just my shoddy formatting.

[henry2cox]

... that's likely being added by the script I used to merge multiple lcov files together

lcov -o aggregate.info -a foo.info -a bar.info -a 'glob*match*.info' ... can be used to merge info files (and might be safer).

Parallelism is disabled by default (--parallel 1) - so that's not it.

[chris-wickens]

Interesting, I didn't know about --add-tracefile. I'm using a Flutter monorepo setup with lcov.info files spread across multiple directories, each lcov.info file path is generated using only a partial path (from the root of the Flutter package the file resides in). When I run the aggregate command from the project root for each lcov.info file the remainder of the file path (from the project root to the package root) is not prepended, so genhtml is unable to find the files and fails with the same error. I suspect the need to combine lcov.info files that were generated with different working directories is why tools like combine_coverage exist, but perhaps you can clarify?

I managed to reproduce the issue on a smaller project using combine_coverage. The generated lcov.info looks correct to me, but since it contains the full path you would need to regenerate it before you can run genhtml, e.g.

dart pub global activate combine_coverage
combine_coverage --repo-path=$(pwd)
genhtml coverage/lcov.info -o coverage_html/html

(Or you could just update the file paths manually)

I tried modifying lcov.info to use relative paths but that didn't help. combine_coverage requires the full repo path to be specified so I'm assuming absolute paths are intended to work.

Reading tracefile coverage/lcov.info.
Found 2 entries.
Found common filename prefix "/Users/chriswickens/bugs/coverage_issue"
Generating output.
Processing file apps/coverage_issue/lib/main.dart
  lines=4 hit=2
 is not readable or doesn't exist. See the '--substitute' and '--synthesize-missing' genhtml options for methods to fix the missing path or ignore the problem.

[henry2cox]

I tried your example locally...but don't see a problem. Downloaded your project as zip, unzipped, then 'cd' into the 'coverage_issue-main' directory, then:

$ ~/work/lcov/bin/genhtml coverage/lcov.info -o foo --substitute "s#/Users/chriswickens/bugs/coverage_issue#\pwd`#" Reading tracefile coverage/lcov.info. Found 2 entries. Found common filename prefix "/home/hcox/coverage_issue-main" Generating output. Processing file apps/coverage_issue/lib/main.dart lines=4 hit=2 Processing file packages/domain/lib/foo.dart lines=1 hit=1 Overall coverage rate: source files: 2 lines.......: 60.0% (3 of 5 lines) functions...: no data found `

My sandbox tool version contains a number of not-yet-pushed changes which may or may not affect what I see. With a bit of luck: the majority of those changes will be pushed "soon".

With respect to combine_coverage or lcov_viewer: I fear that I know nothing at all about either. My suspicion is that the former is merging subdirectory info data while appending parent directory prefixes - but I don't know that for a fact. (Something similar can be done with lcov --add-tracefile and --substitute - but it would take multiple steps.

[henry2cox]

forgot to mention that relative path is also supposed to work: $ ~/work/lcov/bin/genhtml coverage/lcov.info -o foo --substitute "s#/Users/chriswickens/bugs/coverage_issue/##" Reading tracefile coverage/lcov.info. Found 2 entries. Found common filename prefix "/home/hcox/coverage_issue-main" Generating output. Processing file apps/coverage_issue/lib/main.dart lines=4 hit=2 Processing file packages/domain/lib/foo.dart lines=1 hit=1 Overall coverage rate: source files: 2 lines.......: 60.0% (3 of 5 lines) functions...: no data found

[chris-wickens]

Thanks for testing. I'm seeing the issue on my local machine and CI build, so perhaps there's a subtle issue with M1 Macs, wouldn't be the first time. Once your changes are pushed I'll give it another try. Appreciate your help!

[henry2cox]

Should be fixed in 23d166a1742 - pushed this morning. Please go ahead and close this issue if the fix seems sufficient. Thanks Henry

[chris-wickens]

That's fixed it on my local machine at least. Thanks Henry!