Open testQx opened 11 months ago
Trying the following method so far does not generate the relevant *.profraw
cargo build # Build rust binaries. # Commands using binaries in target/debug/*, including `cargo test` and other cargo subcommand. #...
After you built the binaries, did you run them? If you have not run any, no profiles will be generated. If you run the binary, how exactly did you do it? (When using tools that do not propagate environment variables or parent directory structures, such as docker for example, environment variables must be propagated/mapped properly. See also https://github.com/taiki-e/cargo-llvm-cov/issues/319#issuecomment-1750813090)
Trying the following method so far does not generate the relevant *.profraw
cargo build # Build rust binaries. # Commands using binaries in target/debug/*, including `cargo test` and other cargo subcommand. #...
After you built the binaries, did you run them? If you have not run any, no profiles will be generated. If you run the binary, how exactly did you do it? (When using tools that do not propagate environment variables or parent directory structures, such as docker for example, environment variables must be propagated/mapped properly. See also #319 (comment))
My steps are as follows:
But there is no *.profraw file
- kill -9 Indicates related service processes
AFAIK, SIGKILL is currently not supported. See https://github.com/taiki-e/cargo-llvm-cov/issues/235 for more.
Trying the following method so far does not generate the relevant *.profraw
cargo build # Build rust binaries. # Commands using binaries in target/debug/*, including `cargo test` and other cargo subcommand. #...
After you built the binaries, did you run them? If you have not run any, no profiles will be generated. If you run the binary, how exactly did you do it? (When using tools that do not propagate environment variables or parent directory structures, such as docker for example, environment variables must be propagated/mapped properly. See also #319 (comment))
I found a problem, rust services run on physical machines But I mount the working directory through docker and do the steps 1,2,3 above, and there is no cargo command on the physical machine. Then I perform Step 4 on the physical machine. Do environment variables need to be set on the physical machine to run the program? Whether the environment variable set here is the same as the docker internal, using a relative path, or the external physical machine absolute path address
- kill -9 Indicates related service processes
AFAIK, SIGKILL is currently not supported. See #235 for more.
*.profraw file
Is the .profraw file generated when I run pytest (while the rust program is still working), and how do I trigger the .profraw file
Do environment variables need to be set on the physical machine to run the program? Whether the environment variable set here is the same as the docker internal, using a relative path, or the external physical machine absolute path address
set-env currently sets the following environment variables:
export RUSTFLAGS='-C instrument-coverage --cfg=coverage --cfg=trybuild_no_target'
export LLVM_PROFILE_FILE='/path/to/target/name-%p-%10m.profraw'
export CARGO_LLVM_COV=1
export CARGO_LLVM_COV_SHOW_ENV=1
export CARGO_LLVM_COV_TARGET_DIR=/path/to/target
The environment variables actually used in each step are different.
cargo build
requires RUSTFLAGS
.LLVM_PROFILE_FILE
.cargo llvm-cov report
and cargo llvm-cov clean
require CARGO_LLVM_COV_TARGET_DIR
.When using docker or other, absolute paths may need to be mapped to point to the same directory. (LLVM_PROFILE_FILE or/and CARGO_LLVM_COV_TARGET_DIR, depending on the situation)
Is the .profraw file generated when I run pytest (while the rust program is still working), and how do I trigger the .profraw file
IIRC, rust binary (or cdylib) will generate *.profraw file on exit. The LLVM documentation says there is a way to generate file on a regular basis, but it didn't work when I tried it before. (https://github.com/taiki-e/cargo-llvm-cov/pull/280)
Do environment variables need to be set on the physical machine to run the program? Whether the environment variable set here is the same as the docker internal, using a relative path, or the external physical machine absolute path address
set-env currently sets the following environment variables:
export RUSTFLAGS='-C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' export LLVM_PROFILE_FILE='/path/to/target/name-%p-%10m.profraw' export CARGO_LLVM_COV=1 export CARGO_LLVM_COV_SHOW_ENV=1 export CARGO_LLVM_COV_TARGET_DIR=/path/to/target
The environment variables actually used in each step are different.
cargo build
requiresRUSTFLAGS
.- Binaries require
LLVM_PROFILE_FILE
.cargo llvm-cov report
andcargo llvm-cov clean
requireCARGO_LLVM_COV_TARGET_DIR
.When using docker or other, absolute paths may need to be mapped to point to the same directory. (LLVM_PROFILE_FILE or/and CARGO_LLVM_COV_TARGET_DIR, depending on the situation)
Is the .profraw file generated when I run pytest (while the rust program is still working), and how do I trigger the .profraw file
IIRC, rust binary (or cdylib) will generate *.profraw file on exit. The LLVM documentation says there is a way to generate file on a regular basis, but it didn't work when I tried it before. (#280)
I tried using LLVM_PROFILE_FILE, but it didn't work. If I inject so file at compile time, this so file is mainly used to listen to semaphore, can I generate *.profraw file, which method can I trigger internally to generate?
I have hit a similar issue, in this CI run.
As can be seen there, cargo llvm-cov show-env
sets LLVM_PROFILE_FILE
to '/home/runner/work/nearcore/nearcore/target/nearcore-%p-%16m.profraw'
. Yet cargo llvm-cov report
is searching in another folder:
warning: not found *.profraw files in /home/runner/work/nearcore/nearcore/target/llvm-cov-target
I thought this might be a bug in show-env
, and I tried solving it by adding this line to CI, after the show-env:
- run: echo "LLVM_PROFILE_FILE=$(pwd)/target/llvm-cov-target/nearcore-%p-%16m.profraw" >> "$GITHUB_ENV"
However, this resulted in this other failing CI job, where the profraw was still not detected.
I then tried to add back the clean
step, that I was trying to avoid in order to avoid spending tens of minutes of CI time rebuilding dependencies all the time. It still failed, and find
found no *.profraw
files.
The workflow I’m currently trying to get to work is the following: https://github.com/near/nearcore/pull/10180/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR79-R102
Do you have any idea what I’m doing wrong? AFAICT with the last iteration, I’m basically only trying to get the cargo llvm-cov show-env
-based example to work, without anything too specific to our setup.
@Ekleog-NEAR One of the problems is that you are passing the output of --export-prefix to GITHUB_ENV; -export-prefix adds a prefix for the UNIX shell, which is not the format GITHUB_ENV expects.
One of the ways is to save the output of --export-prefix to a file and source
it, as in https://github.com/nextest-rs/nextest/issues/1036.
Ooooh now I feel stupid for copy-pasting without checking the documentation ._. Thank you for the solution! I ended up with this code for setting the environment in a github action. Would you be interested in a PR that showcases the more "advanced" (multiple codecov labels, using with python tests that call into rust, etc.) use cases of cargo-llvm-cov in the readme?
Also I’m curious, does llvm-cov actually need the whole target folder, or would it be possible to trim it somehow, to be able to share build artifacts across jobs without having to share the whole target folder? And if it’s possible (and significantly reduces the size of the target folder), would you be interested in a subcommand that cleans target to just leave the necessary stuff?
Also, just to say, I had parsed --export-prefix
as "also export the prefix, like the target/ directory", resulting in this mistake. Maybe on the next major upgrade of cargo-llvm-cov, you could consider renaming it into --with-export-prefix
, that does not have this potential for mis-parsing? (I’d totally understand if the churn would be too big, it’s just an idea)
Would you be interested in a PR that showcases the more "advanced" (multiple codecov labels, using with python tests that call into rust, etc.) use cases of cargo-llvm-cov in the readme?
A PR to add an example is welcome!
Also I’m curious, does llvm-cov actually need the whole target folder, or would it be possible to trim it somehow, to be able to share build artifacts across jobs without having to share the whole target folder? And if it’s possible (and significantly reduces the size of the target folder), would you be interested in a subcommand that cleans target to just leave the necessary stuff?
Only the binaries and profraw are actually needed to generate the report. I think it would be okay to add a subcommand that copies some of the files that clean_partial deletes instead of deleting and outputs to a specified directory and an option to get the needed files from the directory containing the output of that command instead of the target directory.
Also, just to say, I had parsed
--export-prefix
as "also export the prefix, like the target/ directory", resulting in this mistake. Maybe on the next major upgrade of cargo-llvm-cov, you could consider renaming it into--with-export-prefix
, that does not have this potential for mis-parsing? (I’d totally understand if the churn would be too big, it’s just an idea)
I have no strong opinion on the naming. That said, it may indeed be clearer to clarify that "export" is a noun, not a verb.
One scenario I have is a program written in rust that exposes the http interface and does not generate Python-related dependency packages via pyo3. Invoke http interface directly by executing request through pytest;
Is it possible for me to get a coverage report for this integration test?
Trying the following method so far does not generate the relevant *.profraw
In simple terms, for example, rust program is a back-end, it has a front-end page, can I get the coverage of the relevant operations through the front-end operations