search

cooperwalbrun / rust-template

A project template for setting up new Rust 2021 Edition projects with cargo-make/rustfmt/rust-clippy.
GNU General Public License v3.0
10 stars 2 forks source link

Code Coverage Consistency between Linux and Windows #9

Closed cooperwalbrun closed 2 years ago

cooperwalbrun commented 2 years ago

I noticed that the code coverage report is not equivalent between Linux (ubuntu-latest, i.e. the GitHub Actions environment) and Windows (my machine, i.e. Windows 10). Below, you will see the difference between the two (output generated using cargo make test-coverage).

Windows

Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover Branches Missed Branches Cover
src/main.rs 5 1 80.00% 4 1 75.00% 8 3 62.50% 0 0 -
TOTAL 5 1 80.00% 4 1 75.00% 8 3 62.50% 0 0 -

Linux

Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover Branches Missed Branches Cover
src/main.rs 5 5 0.00% 4 4 0.00% 8 8 0.00% 0 0 -
TOTAL 5 5 0.00% 4 4 0.00% 8 8 0.00% 0 0 -

Details

For reference, the approach taken to gather coverage/profiling data is documented here in the rustc book. There are two scripts in the repository for facilitating the jq and llvm-cov/cargo cov parts of the workflow (one for Linux, one for Windows) but these scripts should have no bearing on the coverage reporting discrepancies. I believe it has to do with how the code gets instrumented, or how the profiler analyzes the code.

Based on the tables above, on Windows it appears that the tooling believes our assert_eq!(true, true) test is actually hitting some part of the (very trivial) main(). Currently, I am not sure I understand why this is the case. To be honest, the numbers generally confound me, as the report claims that there are 5 functions, when the only thing that exists in our program is a one-line main()...

Anyway, I will do more research/investigation and comment on this issue with my findings.

cooperwalbrun commented 2 years ago

It is very possible that the resolution of rust-lang/rust#90132 will address this discrepancy. I do not know enough to say whether this is a side effect of the experimental/nightly nature of the coverage tooling, but nevertheless that issue directly impacts this one so I will keep track of it.

cooperwalbrun commented 2 years ago

Update - the builds triggered after the changes in #11 appear to still have this discrepancy (comparing the GitHub runner with my local Windows 10 still). This confirms that whatever is causing this is not related to the stability of the instrument-coverage feature discussed in rust-lang/rust#90132.

cooperwalbrun commented 2 years ago

I am going to investigate using cargo-llvm-cov instead of the current duct-taped solution (which works just fine aside from the problems described in this issue); outsourcing this complexity seems easier to maintain.

It is possible that performing such refactoring will resolve this issue.

cooperwalbrun commented 2 years ago

This was fixed by #12.