Open epage opened 6 years ago
See also the discussion at https://github.com/assert-rs/assert_cli/issues/104
I'm exactly in this situation right now, and unfortunately, I have no idea how to fix it. I've tried kcov, gcov and tarpaulin without success so far. I'm afraid I may have to go back to testing my cli tool directly by calling directly the main function from my crate. I really like the idea to test the final executable, in order to make sure to not miss anything, and assert_cmd makes is quite pleasant to do, so this is a bit disappointing!
I've been able to get kcov to report something by calling kcov instead of calling directly my program.
I activate it with the kcov
feature in my project. kcov doesn't seem to be able to run in parallel, so I force the test to run on a single thread:
cargo test --features kcov -- --test-threads 1
It would be nice to have something similar natively, for example with an environment variable that allow the developer to set a wrapper — in that case, kcov. Something like
RUST_ASSERT_CMD_WRAPPER="kcov --include-pattern=/src --exclude-pattern=/.cargo $PWD/cov" cargo test -- --test-threads 1
A wrapper sounds like a great way to resolve this since the coverage tools don't follow fork
(granted, when running cargo
would we want them to)
I'm assuming I'll need to use shlex
to separate out the command from the args when calling Command::new
I’ll add that this would be a very nice feature. I’m currently using assert_cmd and tarpaulin and just like others have noted that this combination doesn’t work.
grcov (and kcov) appears to work with assert_cmd. Other tools failed to mark lines as tested as others have mentioned here.
My project doesn't use assert_cmd
, but based on my investigations of kcov
I am pretty sure that special support will need to be added to properly track coverage in subcommand executions. You can check out tectonic-typesetting/tectonic#638 to see my approach:
assert_cmd
-type activities into one harness (called executable
in my project)kcov
, but not doing anything special when launching subcommands (I use cargo kcov
which currently requires some hacking, but that's a separate issue)executable
harness, rerun it outside of kcov
, but with a special environment variable that causes it to launch the underlying tool inside kcov
; it puts the coverage data into a mkdtemp
-type directory name so that runs don't stop on each other's toeskcov --merge
Is there any plan on having this solved? Like @maxcountryman , I've seen that it doesn't work with tarpaulin coverage reports.
Sorry for the delay (Texas storm and life).
I am not actively working on this but would welcome contributions from others.
@brandonkal I've been unable to get grcov to recognize code coverage from assert_cmd. Are you positive that it did work? Was there something special you did?
I am currently able to get both cargo-llvm-cov and grcov to recognize coverage of crate binaries tested with assert_cmd
(though it doesn't work for extra-crate binaries built via test-binary
that use the crate under test). For cargo-llvm-cov, I simply ran cargo llvm-cov --all-features --lcov --output-path lcov.info
, and for grcov
, I ran:
export RUSTFLAGS="${RUSTFLAGS:+$RUSTFLAGS }-Cinstrument-coverage"
cargo build
export LLVM_PROFILE_FILE="target/coverage/prof/%p-%m.profraw"
cargo test --all-features
grcov \
--source-dir . \
--binary-path target/debug \
--branch \
--excl-start 'mod tests \{' \
--ignore 'tests/*' \
-t html \
-o target/coverage/html \
target/coverage/prof
An example repository where you can see coverage working: https://github.com/jwodder/ghrepo-rust
For those that stumble across this issue because they can't find a solution on how to get cargo-llvm-cov to work:
I got it working by using the command from above (cargo llvm-cov --all-features --lcov --output-path lcov.info
) and by placing my test file into a seperate tests/
folder instead of src/some_file.rs
.
@mssun
From https://github.com/rust-lang-nursery/cli-wg/issues/9#issuecomment-385487756