cargo-llvm-cov doesn't report results with crate inside tests directory

[xd009642]

So I'm integrating an llvm coverage-instrumentation based backend to tarpaulin and had a few false negatives compared to my ptrace implementation on one of my projects used to test coverage. So checked it in cargo-llvm-cov and it picked up 0 lines. My implementation using -Cinstrument-coverage picked up the files and all the same lines as ptrace implementation except for two libc::exit calls. I'll try using just the RUSTFLAGS myself and profdata and cov binaries directly after work. But as this is my first time trying cargo-llvm-cov to avoid that faff I wondered if I was doing anything wrong.

https://github.com/xd009642/tarpaulin/tree/develop/tests/data/fork-test

The output:

cargo llvm-cov
   Compiling libc v0.2.104
   Compiling fork-test v0.1.0 (/home/daniel/personal/tarpaulin/tests/data/fork-test)
    Finished test [unoptimized + debuginfo] target(s) in 1.54s
     Running unittests src/lib.rs (target/llvm-cov-target/debug/deps/fork_test-cee38c9777d8078f)

running 3 tests
test test1 ... ok
test test2 ... ok
test test3 ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Filename                      Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                               0                 0         -           0                 0         -           0                 0         -           0                 0         -

Also tarpaulin output for comparison:

cargo tarpaulin --engine llvm -r tests/data/fork-test/`
Jul 01 12:13:46.283  INFO cargo_tarpaulin::config: Creating config
Jul 01 12:13:46.303  INFO cargo_tarpaulin: Running Tarpaulin
Jul 01 12:13:46.303  INFO cargo_tarpaulin: Building project
Jul 01 12:13:46.303  INFO cargo_tarpaulin::cargo: Cleaning project
   Compiling libc v0.2.104
   Compiling fork-test v0.1.0 (/home/daniel/personal/tarpaulin/tests/data/fork-test)
    Finished test [unoptimized + debuginfo] target(s) in 1.67s
  Executable unittests src/lib.rs (tests/data/fork-test/target/debug/deps/fork_test-d95f40770f51fc1c)
Jul 01 12:13:48.110  INFO cargo_tarpaulin::process_handling: running /home/daniel/personal/tarpaulin/tests/data/fork-test/target/debug/deps/fork_test-d95f40770f51fc1c
Jul 01 12:13:48.110  INFO cargo_tarpaulin::process_handling: Setting LLVM_PROFILE_FILE

running 3 tests
test test1 ... ok
test test2 ... ok
test test3 ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Jul 01 12:13:49.616  INFO cargo_tarpaulin::statemachine::instrumented: For binary: target/debug/deps/fork_test-d95f40770f51fc1c
Jul 01 12:13:49.616  INFO cargo_tarpaulin::statemachine::instrumented: Generated: fork_test-d95f40770f51fc1c_14148.profraw
Jul 01 12:13:49.623  INFO cargo_tarpaulin::report: Coverage Results:
|| Uncovered Lines:
|| src/lib.rs: 24, 37
|| Tested/Total Lines:
|| src/lib.rs: 14/16 -6.25%
|| 
87.50% coverage, 14/16 lines covered, -6.25% change in coverage

And for completeness what the fork test looks like (only dep is libc="0.2.103"

#[test]
pub fn test1()  {
    fn child<F: FnOnce()>(f: F) {
        match unsafe { libc::fork() } {
           0 => {
               f();
               unsafe {
                   libc::_exit(0);
               }
           },
           -1 => unreachable!(),
           pid => unsafe {
               libc::waitpid(pid, core::ptr::null_mut(), 0);
           },
        }
    }

    child(|| ());
}

#[test]
pub fn test2()  {
    match unsafe { libc::fork() } {
       0 => unsafe { libc::_exit(0); }, // line 24
       -1 => unreachable!(),
       pid => unsafe {
           libc::waitpid(pid, core::ptr::null_mut(), 0);
       },
    }
}

#[test]
pub fn test3()  {
    match unsafe { libc::fork() } {
       0 => {
           unsafe {
               libc::_exit(0); // line 37
           }
       },
       -1 => unreachable!(),
       pid => unsafe {
           libc::waitpid(pid, core::ptr::null_mut(), 0);
       },
    }
}

[taiki-e]

Thanks for the report! This seems similar to https://github.com/rust-lang/rust/issues/77553, but I couldn't reproduce this on ubuntu 20.04. Could you provide information on the environment in which this occurred?

FYI, below is the environment in which I ran the test:

$ uname -a
Linux parallels-Parallels-Virtual-Platform 5.13.0-44-generic #49~20.04.1-Ubuntu SMP Wed May 18 18:44:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ rustc -vV
rustc 1.61.0 (fe5b13d68 2022-05-18)
binary: rustc
commit-hash: fe5b13d681f25ee6474be29d748c65adcd91f69e
commit-date: 2022-05-18
host: x86_64-unknown-linux-gnu
release: 1.61.0
LLVM version: 14.0.0

$ cargo llvm-cov --version
cargo-llvm-cov 0.4.8

$ cargo tree
repro v0.1.0 (/home/parallels/projects/tmp/repro)
└── libc v0.2.126

$ cargo llvm-cov          
   Compiling repro v0.1.0 (/home/parallels/projects/tmp/repro)
    Finished test [unoptimized + debuginfo] target(s) in 0.31s
     Running unittests src/lib.rs (target/llvm-cov-target/debug/deps/repro-7281c5c75dd71403)

running 3 tests
test test1 ... ok
test test2 ... ok
test test3 ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Filename                                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/home/parallels/projects/tmp/repro/src/lib.rs          19                 7    63.16%           9                 1    88.89%          52                 9    82.69%           0                 0         -
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                  19                 7    63.16%           9                 1    88.89%          52                 9    82.69%           0                 0         -

[taiki-e]

Another possibility is that the regex used here is a bit rough; if passing --remap-path-prefix or --disable-default-ignore-filename-regex solves the problem, I think that is the cause.

[xd009642]

uname -a
Linux daniel-XPS-15-9550 5.4.0-120-generic #136~18.04.1-Ubuntu SMP Fri Jun 10 18:00:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

rustc -vV
rustc 1.61.0 (fe5b13d68 2022-05-18)
binary: rustc
commit-hash: fe5b13d681f25ee6474be29d748c65adcd91f69e
commit-date: 2022-05-18
host: x86_64-unknown-linux-gnu
release: 1.61.0
LLVM version: 14.0.0

cargo-llvm-cov 0.4.8

And I just tried with --disable-default-ignore-filename-regex and got coverage results, so that must be it! FYI on tarpaulin, I use walk-dir and cargo-metadata to remove target dir, CARGO_HOME, vendored sources and keep it to just the source code. Foudn that was more robust than my initial regex attempts (people setting their CARGO_HOME to inside project directory and things in build scripts add a lot of interesting edge cases)