The llvm-cov command accidentally produces empty coverage reports if the filename regex matches something in the local path:
tools/generate_test_coverage_report.sh.
Current Behavior
In the script tools/generate_test_coverage_report.sh the command contains a filename regex to exclude certain files out of the coverage report.
The command looks like this:
RUST_LOG=debug cargo llvm-cov --ignore-filename-regex "test|main|test_utils|cli.rs" "$@"
But the "--ignore-filename-regex" option behaves like an "--ignore-filepath-regex", so whenever a substring in the complete absolute path matches the regex the whole repo is excluded from the coverage scan.
If someone pulls the Ankaios Repo for local testing and put it inside a folder "ankaios_test" the user would get an empty coverage report because "test" is inside the regex filter for the path /home/.../ankaios_test/ankaios/....
This is dangerous, because implementation files can be accidentally excluded as well.
The main problem is, it is a valid run. You wouldn't receive any error message or trace (tried with enabling different trace levels and verbose cli options (-vvv)). Just an empty report is generated.
Expected Behavior
The regex expressions shall be made more explicit for exclusions.
The documentation and the name of the cli option indicates more that it matches only against file names or at least the relative path starting inside the repo, instead of the whole file path. This would feel more naturally to not have a dependency of the filters on the parent paths or the repository name itself.
Because this means that a successfully generated coverage report depends on how the user who downloads Ankaios repo names the parts or subfolders of the parent paths.
Steps to Reproduce
While I have found out this confusing behavior when implementing CI/CD for Ankaios Repo,
it can be easiliy reproduced with the following steps:
Open a shell in VSCode in the development container
Copy the ankaios repo into another path containing a folder name matching something in the regex above: mkdir /home/vscode/ankaios_test
cp -r ankaios /home/vscode/ankaios_test/ # here "ankaios_test" matches the filter regex "test" in the llvm-cov command
cd /home/vscode/ankaios_test/ankaios
tools/generate_test_coverage_report.sh test --html # run the coverage report in the newly copied repo
ls /home/vscode/ankaios_test/ankaios/target/llvm-cov/html
With step 6 you can see that no sub folder "coverage" is created containing all detailed htmls for agent, ank, server and so on.
With step 7 you can check the global html report showing an empty report (only 1 row in the table).
Context (Environment)
Reproducible in the devcontainer in VSCode.
Logs
Missing "coverage" subfolder and empty or small index.html:
Additional Information
You can make it work again with just renaming the ankios_test folder that it does not match against something inside the filter regex above: mv ankaios_test ankaiosdev
Then running the coverage script again produces the correct report again:
Note: The fact that it repeats the absolute path inside the "coverage" subfolder has given me the hint that it matches against the whole path not just the filename or the relative path starting inside the repo.
Reported by @inf17101:
The llvm-cov command accidentally produces empty coverage reports if the filename regex matches something in the local path:
tools/generate_test_coverage_report.sh
.Current Behavior
In the script
tools/generate_test_coverage_report.sh
the command contains a filename regex to exclude certain files out of the coverage report.The command looks like this:
RUST_LOG=debug cargo llvm-cov --ignore-filename-regex "test|main|test_utils|cli.rs" "$@"
The documentation of llvm-cov (https://github.com/taiki-e/cargo-llvm-cov#exclude-file-from-coverage) says that the regex can be used to exclude some files.
But the "--ignore-filename-regex" option behaves like an "--ignore-filepath-regex", so whenever a substring in the complete absolute path matches the regex the whole repo is excluded from the coverage scan.
If someone pulls the Ankaios Repo for local testing and put it inside a folder "ankaios_test" the user would get an empty coverage report because "test" is inside the regex filter for the path /home/.../ankaios_test/ankaios/....
This is dangerous, because implementation files can be accidentally excluded as well.
The main problem is, it is a valid run. You wouldn't receive any error message or trace (tried with enabling different trace levels and verbose cli options (-vvv)). Just an empty report is generated.
Expected Behavior
The regex expressions shall be made more explicit for exclusions. The documentation and the name of the cli option indicates more that it matches only against file names or at least the relative path starting inside the repo, instead of the whole file path. This would feel more naturally to not have a dependency of the filters on the parent paths or the repository name itself.
Because this means that a successfully generated coverage report depends on how the user who downloads Ankaios repo names the parts or subfolders of the parent paths.
Steps to Reproduce
While I have found out this confusing behavior when implementing CI/CD for Ankaios Repo, it can be easiliy reproduced with the following steps:
With step 6 you can see that no sub folder "coverage" is created containing all detailed htmls for agent, ank, server and so on. With step 7 you can check the global html report showing an empty report (only 1 row in the table).
Context (Environment)
Reproducible in the devcontainer in VSCode.
Logs
Missing "coverage" subfolder and empty or small index.html:
Additional Information
You can make it work again with just renaming the ankios_test folder that it does not match against something inside the filter regex above: mv ankaios_test ankaiosdev
Then running the coverage script again produces the correct report again:
Note: The fact that it repeats the absolute path inside the "coverage" subfolder has given me the hint that it matches against the whole path not just the filename or the relative path starting inside the repo.