Coverage reporting missing for binaries under-test

epage commented 6 years ago

@mssun

Another issue about testing CLIs is to correctly calculate code coverage. If using std::process::Command to fork and execute a CLI, all existing code coverage tools (tarpaulin, gcov, kcov, etc) will failed to calculate the statistics. There is not a better way to properly calculate the line coverage including unit tests and integration tests.

From https://github.com/rust-lang-nursery/cli-wg/issues/9#issuecomment-385487756

epage commented 6 years ago

See also the discussion at https://github.com/assert-rs/assert_cli/issues/104

glehmann commented 5 years ago

I'm exactly in this situation right now, and unfortunately, I have no idea how to fix it. I've tried kcov, gcov and tarpaulin without success so far. I'm afraid I may have to go back to testing my cli tool directly by calling directly the main function from my crate. I really like the idea to test the final executable, in order to make sure to not miss anything, and assert_cmd makes is quite pleasant to do, so this is a bit disappointing!

glehmann commented 5 years ago

I've been able to get kcov to report something by calling kcov instead of calling directly my program.

https://github.com/glehmann/hld/blob/f40f3b51f84969cc13714f81451ad17f33fbf2dc/tests/common/mod.rs#L79

I activate it with the kcov feature in my project. kcov doesn't seem to be able to run in parallel, so I force the test to run on a single thread:

cargo test --features kcov -- --test-threads 1

It would be nice to have something similar natively, for example with an environment variable that allow the developer to set a wrapper — in that case, kcov. Something like

RUST_ASSERT_CMD_WRAPPER="kcov --include-pattern=/src --exclude-pattern=/.cargo $PWD/cov" cargo test  -- --test-threads 1

epage commented 5 years ago

A wrapper sounds like a great way to resolve this since the coverage tools don't follow fork (granted, when running cargo would we want them to)

epage commented 5 years ago

I'm assuming I'll need to use shlex to separate out the command from the args when calling Command::new

maxcountryman commented 4 years ago

I’ll add that this would be a very nice feature. I’m currently using assert_cmd and tarpaulin and just like others have noted that this combination doesn’t work.

brandonkal commented 4 years ago

grcov (and kcov) appears to work with assert_cmd. Other tools failed to mark lines as tested as others have mentioned here.

pkgw commented 4 years ago

My project doesn't use assert_cmd, but based on my investigations of kcov I am pretty sure that special support will need to be added to properly track coverage in subcommand executions. You can check out tectonic-typesetting/tectonic#638 to see my approach:

Gather the tests that do assert_cmd-type activities into one harness (called executable in my project)
Run the test suite inside kcov, but not doing anything special when launching subcommands (I use cargo kcov which currently requires some hacking, but that's a separate issue)
For the executable harness, rerun it outside of kcov, but with a special environment variable that causes it to launch the underlying tool inside kcov; it puts the coverage data into a mkdtemp-type directory name so that runs don't stop on each other's toes
Merge all of the coverage results together at the end with kcov --merge

frosklis commented 3 years ago

Is there any plan on having this solved? Like @maxcountryman , I've seen that it doesn't work with tarpaulin coverage reports.

epage commented 3 years ago

Sorry for the delay (Texas storm and life).

I am not actively working on this but would welcome contributions from others.

milesj commented 2 years ago

@brandonkal I've been unable to get grcov to recognize code coverage from assert_cmd. Are you positive that it did work? Was there something special you did?

jwodder commented 1 year ago

I am currently able to get both cargo-llvm-cov and grcov to recognize coverage of crate binaries tested with assert_cmd (though it doesn't work for extra-crate binaries built via test-binary that use the crate under test). For cargo-llvm-cov, I simply ran cargo llvm-cov --all-features --lcov --output-path lcov.info, and for grcov, I ran:

export RUSTFLAGS="${RUSTFLAGS:+$RUSTFLAGS }-Cinstrument-coverage"
cargo build
export LLVM_PROFILE_FILE="target/coverage/prof/%p-%m.profraw"
cargo test --all-features

grcov \
    --source-dir . \
    --binary-path target/debug \
    --branch \
    --excl-start 'mod tests \{' \
    --ignore 'tests/*' \
    -t html \
    -o target/coverage/html \
    target/coverage/prof

An example repository where you can see coverage working: https://github.com/jwodder/ghrepo-rust

LMH01 commented 1 year ago

For those that stumble across this issue because they can't find a solution on how to get cargo-llvm-cov to work:

I got it working by using the command from above (cargo llvm-cov --all-features --lcov --output-path lcov.info) and by placing my test file into a seperate tests/ folder instead of src/some_file.rs.

assert-rs / assert_cmd

Coverage reporting missing for binaries under-test #9