Closed TriplEight closed 1 year ago
Hmm, this is odd because we already ignore --profile option on nextest.
Could you check does https://github.com/taiki-e/cargo-llvm-cov/commit/ebd846b97608ee8a8c5c5a31216f8f78bd5b0f9c (in nextest-profile branch) fixes this?
(Also please remove run
after cargo llvm-cov nextest
.)
Thanks for the quick reply! I ran
NEXTEST_PROFILE="scripts/ci/gitlab/nextest.toml"
cargo install --git https://github.com/taiki-e/cargo-llvm-cov cargo-llvm-cov --rev ebd846b97608ee8a8c5c5a31216f8f78bd5b0f9c --force --locked
cargo llvm-cov nextest
--workspace
--cargo-profile testnet
--verbose
--locked
--features=runtime-benchmarks,runtime-metrics
--no-fail-fast
--color always
--verbose
--include-build-script
--html
--output-dir $ARCHIVE
and got
Running `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --cargo-profile testnet --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v`
error: profile `scripts/ci/gitlab/nextest.toml not found (known profiles: default, default-miri)`
error: process didn't exit successfully: `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --cargo-profile testnet --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v` (exit status: 96)
*removed cargo
from profile
and got the error doubled:
Running `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --profile testnet --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v`
error: profile `testnet not found (known profiles: default, default-miri)`
error: process didn't exit successfully: `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --profile testnet --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v` (exit status: 96)
Running `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --profile testnet --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v`
error: profile `testnet not found (known profiles: default, default-miri)`
error: process didn't exit successfully: `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --profile testnet --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v` (exit status: 96)
Hmm... do you have the same problem when running nextest without cargo-llvm-cov?
I guess that it has nothing to do with cargo-llvm-cov, since cargo-llvm-cov does not look at the NEXTEST_PROFILE environment variable and does not change the current directory. (At least in cases where scripts/ci/gitlab/nextest.toml is shown in error.)
No, with nextest I run it normally with --cargo-profile scripts/ci/gitlab/nextest.toml
. (Maybe the problem is that you're passing the profile
and not cargo-profile
here)
I passed NEXTEST_PROFILE
as a possible workaround, because couldn't find a suitable option in cargo-llvm
.
I run it normally with
--cargo-profile scripts/ci/gitlab/nextest.toml
.
Um, you seem to be confused about which flag you passed. In logs in your other comments, I did not see any case where you passed --cargo-profile scripts/ci/gitlab/nextest.toml
to cargo-llvm-cov.
(After unset NEXTEST_PROFILE,) try passing the exact same thing you passed to cargo-nextest to cargo-llvm-cov. https://github.com/taiki-e/cargo-llvm-cov/issues/267#issuecomment-1527674997 's patch should be able to handle it.
Maybe the problem is that you're passing the
profile
and notcargo-profile
here
That should work correctly since parse_opt_passthrough refers to the current flag, not the variable name passed. In fact, https://github.com/taiki-e/cargo-llvm-cov/issues/267#issuecomment-1527714964 's log shows that it takes --cargo-profile testnet
as input and runs --cargo-profile testnet
:
--workspace --cargo-profile testnet --locked
I have confirmed that the https://github.com/taiki-e/cargo-llvm-cov/issues/267#issuecomment-1527674997 's patch locally can handle both options. So opened https://github.com/taiki-e/cargo-llvm-cov/pull/269.
Well, the real complication here is that it was --cargo-profile
that was not being processed correctly prior to the https://github.com/taiki-e/cargo-llvm-cov/issues/267#issuecomment-1527674997 's patch, and in fact --profile
is being processed correctly with or without the https://github.com/taiki-e/cargo-llvm-cov/issues/267#issuecomment-1527674997 's patch.
Fixed in v0.5.19
Sorry there was a typo:
No, with nextest I run it normally with
--cargo-profile scripts/ci/gitlab/nextest.toml
. (Maybe the problem is that you're passing the profile and not cargo-profile here)
I run it normally with --profile scripts/ci/gitlab/nextest.toml --cargo-profile testnet
Tested, and can't confirm it's working:
Installed package `cargo-llvm-cov v0.5.19` (executable `cargo-llvm-cov`)
$ cargo llvm-cov nextest --workspace --cargo-profile testnet --profile scripts/ci/gitlab/nextest.toml --verbose --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always --verbose --include-build-script --html --output-dir $ARCHIVE
Running `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --cargo-profile testnet --profile scripts/ci/gitlab/nextest.toml --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v`
error: profile `scripts/ci/gitlab/nextest.toml not found (known profiles: default, default-miri)`
error: process didn't exit successfully: `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --cargo-profile testnet --profile scripts/ci/gitlab/nextest.toml --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v` (exit status: 96)
Hmm, path-based profiles seem to be causing the problem? (I have read the documentation, and it is not even clear that such a profile is supported in the first place.)
cc @sunshowers
You are correct, in https://nexte.st/book/running.html there's
--config-file <PATH>
Config file [default: workspace-root/.config/nextest.toml]
And --profile
reads configured profiles from https://nexte.st/book/configuration.html.
Trying to run with --config-file scripts/ci/gitlab/nextest.toml
Now it's working with
$ cargo llvm-cov nextest --workspace --cargo-profile testnet --config-file scripts/ci/gitlab/nextest.toml --verbose --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always --verbose --include-build-script --html --output-dir $ARCHIVE
[2023-04-29 12:55:12] Running `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --cargo-profile testnet --config-file scripts/ci/gitlab/nextest.toml --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v`
There are no mentions of config-file
in cargo-llvm-cov code. Checking if it's being passed to nextest.
UPD: can confirm --cargo-profile testnet
is working as expected.
Checking if --profile
supports custom profile from --config-file scripts/ci/gitlab/nextest.toml
There are no mentions of config-file in cargo-llvm-cov code.
cargo-llvm-cov doesn't parse all the cargo-nextest CLI flags, but only those necessary for cargo-llvm-cov to work properly, and just passthrough everything else to the cargo-nextest.
This design eliminates the need to respond to each addition or deletion of CLI flags and also makes it easier to maintain compatibility with older cargo-nextest.
Checking if it's being passed to nextest.
It is passed to cargo-nextest. See this part in the log:
--cargo-profile testnet --config-file scripts/ci/gitlab/nextest.toml --verbose
Closing as it has been confirmed that passing the correct option (--config-file) works.
cc https://github.com/taiki-e/cargo-llvm-cov/issues/120
I'm running
And getting an error:
Ran with
--cargo-profile
, like in nextest doc, with the exactly the same result. Same when I removedrun
after nextest.Is there any workaround for it by now?