lib.rs lines miscounted

tcmal commented 2 years ago

Describe the bug

The line count for lib.rs in a crate seems to be miscounted.

To Reproduce

Minimal example here. This contains no actual lines in lib.rs, and only one function in boolean.rs, but the coverage report shows:

|| Tested/Total Lines:
|| src/boolean.rs: 4/4 +0.00%
|| src/lib.rs: 65/259 +0.00%
|| src/main.rs: 5/5 +0.00%
|| 
27.61% coverage, 74/268 lines covered, +0% change in coverage

Looking at the HTML report, no lines are highlighted in lib.rs. cargo llvm-cov correctly reports lib.rs as having 100% coverage with 1 line.

The size of the miscount seems to vary depending on the size of the crate - a larger crate is off by >1000 lines.

Expected behavior

lib.rs should be 1/1 (or possibly 0/0?)in this case.

xd009642 commented 2 years ago

What compiler version are you using because I get:

|| Uncovered Lines:
|| Tested/Total Lines:
|| src/boolean.rs: 4/4 +0.00%
|| src/main.rs: 6/6 +0.00%
|| 
100.00% coverage, 10/10 lines covered, +0% change in coverage

I'd also argue 100% coverage of lib.rs is incorrect as no lines in it are executed, 0 coverable lines and 0 lines covered would be more correct

tcmal commented 2 years ago

I'm using:

rustc 1.63.0 (4b91a6ea7 2022-08-08)
cargo 1.63.0 (fd9c4297c 2022-07-01)
cargo-tarpaulin version: 0.20.0

xd009642 @.***> writes:

What compiler version are you using because I get:

Uncovered Lines: Tested/Total Lines: src/boolean.rs: 4/4 +0.00% src/main.rs: 6/6 +0.00%

100.00% coverage, 10/10 lines covered, +0% change in coverage

I'd also argue 100% coverage of lib.rs is incorrect as no lines in it are executed, 0 coverable lines and 0 lines covered would be more correct

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

xd009642 commented 2 years ago

Nope I get the same output as I pasted above for 1.61, 1.63 and one of the 1.64 nightlies.

Can you give more info about how your system and how you're running it? i.e. OS, gnu or musl libc. Also full --debug output

mihaigalos commented 2 years ago

Line counting is broken, seeing the same behavior as the OP.

$ rust --version
rustc 1.63.0 (4b91a6ea7 2022-08-08)

$ cargo --version
cargo 1.63.0 (fd9c4297c 2022-07-01)

$ cargo tarpaulin --version
cargo-tarpaulin version: 0.18.5

Here's what I get in Codecov, appartently the lib.rs is only 33.79% covered. Also, the newline at the end is "covered" ?!

Screenshot:

mihaigalos commented 2 years ago

Seems the #[test] and #[cfg(test)] get counted as "uncovered" now:

mihaigalos commented 2 years ago

Minimal example here.

In your minimal example, here's what I get:

/tmp/tmp.W3Q1ht5x8y/tarpaulin-issue-example master* » cargo tarpaulin
Aug 15 16:02:46.225  INFO cargo_tarpaulin::config: Creating config                                                                                                                                                   
Aug 15 16:02:46.235  INFO cargo_tarpaulin: Running Tarpaulin                                                                                                                                                         
Aug 15 16:02:46.235  INFO cargo_tarpaulin: Building project
Aug 15 16:02:46.235  INFO cargo_tarpaulin::cargo: Cleaning project
   Compiling coverage-test v0.1.0 (/tmp/tmp.W3Q1ht5x8y/tarpaulin-issue-example)
    Finished test [unoptimized + debuginfo] target(s) in 0.54s
Aug 15 16:02:46.831  INFO cargo_tarpaulin::process_handling::linux: Launching test
Aug 15 16:02:46.831  INFO cargo_tarpaulin::process_handling: running /tmp/tmp.W3Q1ht5x8y/tarpaulin-issue-example/target/debug/deps/coverage_test-0ea23c6ebbe22d0c

running 0 tests                                                                                           

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s                                                                                                                        

Aug 15 16:02:47.187  INFO cargo_tarpaulin::process_handling::linux: Launching test
Aug 15 16:02:47.187  INFO cargo_tarpaulin::process_handling: running /tmp/tmp.W3Q1ht5x8y/tarpaulin-issue-example/target/debug/deps/coverage_test-2d4092681ba255e0

running 1 test
test tests::test_main ... ok                                                                              

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Aug 15 16:02:47.528  INFO cargo_tarpaulin::report: Coverage Results:
|| Tested/Total Lines:
|| src/boolean.rs: 4/4                                                                                    
|| src/lib.rs: 57/259
|| src/main.rs: 5/5                                                                                       
||                 
24.63% coverage, 66/268 lines covered, -75.3731343283582% change in coverage

Strangely enough, the tarpaulin docker latest, published 6h ago, doesn't produce the bug:

» docker run --rm -it --security-opt seccomp=unconfined  -v "${PWD}:/volume" xd009642/tarpaulin
   Compiling coverage-test v0.1.0 (/volume)
    Finished dev [unoptimized + debuginfo] target(s) in 0.53s
Aug 15 14:02:59.092  INFO cargo_tarpaulin::config: Creating config
Aug 15 14:02:59.104  INFO cargo_tarpaulin: Running Tarpaulin
Aug 15 14:02:59.104  INFO cargo_tarpaulin: Building project
Aug 15 14:02:59.104  INFO cargo_tarpaulin::cargo: Cleaning project
   Compiling coverage-test v0.1.0 (/volume)
    Finished test [unoptimized + debuginfo] target(s) in 0.56s
Aug 15 14:02:59.719  INFO cargo_tarpaulin::process_handling::linux: Launching test
Aug 15 14:02:59.719  INFO cargo_tarpaulin::process_handling: running /volume/target/debug/deps/coverage_test-0ea23c6ebbe22d0c

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Aug 15 14:02:59.975  INFO cargo_tarpaulin::process_handling::linux: Launching test
Aug 15 14:02:59.975  INFO cargo_tarpaulin::process_handling: running /volume/target/debug/deps/coverage_test-2d4092681ba255e0

running 1 test
test tests::test_main ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

Aug 15 14:03:00.227  INFO cargo_tarpaulin::report: Coverage Results:
|| Uncovered Lines:
|| Tested/Total Lines:
|| src/boolean.rs: 4/4
|| src/main.rs: 5/5
|| 
100.00% coverage, 9/9 lines covered, +75.3731343283582% change in coverage

mihaigalos commented 2 years ago

@tcmal: cargo install cargo-tarpaulin and rerunning cargo tarpaulin produced the correct 100% coverage for me in your minimal example.

This upgraded to tarpaulin 0.20.1.

xd009642 commented 2 years ago

Ooooh that'll be this change in 0.20.1:

Add max line for each file into source analysis and filter out lines beyond this range #1016

I remember now, rust changed the generated test harness stuff to get debug info lines placed after the end of lib.rs which confused coverage results. Yeah so if you update to 0.20.1 it should be fixed. I'm gonna close this as done but feel free to comment or reopen if it doesn't fix it

xd009642 / tarpaulin

lib.rs lines miscounted #1073