search

rust-lang / rust

Empowering everyone to build reliable and efficient software.
https://www.rust-lang.org
Other
97.96k stars 12.69k forks source link

Output filenames for -Z instrument-coverage #79899

Open vmx opened 3 years ago

vmx commented 3 years ago

I'm trying out the new -Z instrument-coverage support. One issue I ran into that by default the output is stored in a file called default.profraw. That file gets overridden on every new run.

When running cargo test that is a problem. It runs several executables (e.g. for the library tests, integration tests doc tests), where each of them overrides the default.profraw.

I don't know if that is possible, but it would b great to have .profraw files named after the executable they were run with.

Until then what you can do is running cargo test with LLVM_PROFILE_FILE set to %m.profraw:

LLVM_PROFILE_FILE="%m.profraw" CARGO_INCREMENTAL=0 RUSTFLAGS="-Clink-dead-code -Copt-level=0 -Ccodegen-units=1 -Zpanic_abort_tests -Cpanic=abort -Coverflow-checks=off -Zinstrument-coverage -Cinline-threshold=0" cargo +nightly test

Which will then result in file names something like 52458732642933725367_0.profraw, which can then be merged into a single indexed profile file via llvm-profdata merge -o default.profdata *.profraw.

richkadel commented 3 years ago

Since the file is generated by your own target binary (not from a Rust language tool), it's not something the compiler can easily work around.

The default name default.profraw and handling of LLVM_PROFILE_FILE is embedded in the LLVM compiler-rt library, and it is not language specific (it works the same for Clang and Swift, for example). It may be more effective to file a feature request with the LLVM project.

vmx commented 3 years ago

@richkadel cargo test runs several binaries, couldn't cargo test then set LLVM_PROFILE_FILE to something sensible per test run by default and use LLVM_PROFILE_FILE if set by the user?

richkadel commented 3 years ago

That's a good point. All of the work to-date has been on rustc, so I haven't made any changes to cargo and that doesn't normally come to mind. Thanks for the suggested feature!

vmx commented 3 years ago

After playing more with the coverage stuff. I think it would be great if the coverage output could be placed next to the binary it was run with. For example if a test executable is at target/debug/deps/my_test-246fe4d9493aa502, the coverage output could be at target/debug/deps/my_test-246fe4d9493aa502.profraw.

This is useful as the llvm-cov expects to provide a binary with the coverage information in order to produce the report, this way you could easily map the profiling output to the corresponding binary.

richkadel commented 3 years ago

@vmx - Sounds good. I don't know if you'd be interested in working on this feature yourself, but I'd welcome the contribution.

I'm not likely to have time to work on this myself due to shifting work priorities.

If you can and want to submit a PR, please @richkadel mention me for a review (I will send feedback, but I'm not authorized to approve or anything), and add @tmandry and @wesleywiser as well. Either one of them can approve, once ready.

Thanks!

scole66 commented 3 years ago

Might be worth a comment in the document about this problem. I've had -Zinstrument-coverage just bundled into my environment, and I just realized that my coverage broke because I added "cargo install cargo-binutils", which compiled a bunch of things, all with coverage enabled (oops), and every time I run "cargo cov ..." I destroy my profraw files, unwittingly.

taiki-e commented 1 year ago

@rustbot label -requires-nightly

(instrument-coverage has been stabilized)