When used with nextest, `--profile <path>` is not being processed correctly

[TriplEight]

cc https://github.com/taiki-e/cargo-llvm-cov/issues/120

I'm running

NEXTEST_PROFILE="scripts/ci/gitlab/nextest.toml"
cargo llvm-cov nextest run
        --workspace
        --profile testnet
        --verbose
        --locked
        --features=runtime-benchmarks,runtime-metrics
        --no-fail-fast
        --color always
        --verbose
        --include-build-script
        --html
        --output-dir $ARCHIVE

And getting an error:

Running `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target run --workspace --profile testnet --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v`
error: profile `scripts/ci/gitlab/nextest.toml not found (known profiles: default, default-miri)`

Ran with --cargo-profile, like in nextest doc, with the exactly the same result. Same when I removed run after nextest.

Is there any workaround for it by now?

[taiki-e]

Hmm, this is odd because we already ignore --profile option on nextest.

Could you check does https://github.com/taiki-e/cargo-llvm-cov/commit/ebd846b97608ee8a8c5c5a31216f8f78bd5b0f9c (in nextest-profile branch) fixes this?

(Also please remove run after cargo llvm-cov nextest.)

[TriplEight]

Thanks for the quick reply! I ran

NEXTEST_PROFILE="scripts/ci/gitlab/nextest.toml"
cargo install --git https://github.com/taiki-e/cargo-llvm-cov cargo-llvm-cov --rev ebd846b97608ee8a8c5c5a31216f8f78bd5b0f9c --force --locked
cargo llvm-cov nextest
        --workspace
        --cargo-profile testnet
        --verbose
        --locked
        --features=runtime-benchmarks,runtime-metrics
        --no-fail-fast
        --color always
        --verbose
        --include-build-script
        --html
        --output-dir $ARCHIVE

and got

Running `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --cargo-profile testnet --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v`
error: profile `scripts/ci/gitlab/nextest.toml not found (known profiles: default, default-miri)`
error: process didn't exit successfully: `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --cargo-profile testnet --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v` (exit status: 96)

[TriplEight]

*removed cargo from profile and got the error doubled:

Running `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --profile testnet --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v`
error: profile `testnet not found (known profiles: default, default-miri)`
error: process didn't exit successfully: `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --profile testnet --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v` (exit status: 96)

Running `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --profile testnet --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v`
error: profile `testnet not found (known profiles: default, default-miri)`
error: process didn't exit successfully: `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --profile testnet --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v` (exit status: 96)

[taiki-e]

Hmm... do you have the same problem when running nextest without cargo-llvm-cov?

I guess that it has nothing to do with cargo-llvm-cov, since cargo-llvm-cov does not look at the NEXTEST_PROFILE environment variable and does not change the current directory. (At least in cases where scripts/ci/gitlab/nextest.toml is shown in error.)

[TriplEight]

No, with nextest I run it normally with --cargo-profile scripts/ci/gitlab/nextest.toml. (Maybe the problem is that you're passing the profile and not cargo-profile here)

I passed NEXTEST_PROFILE as a possible workaround, because couldn't find a suitable option in cargo-llvm.

[taiki-e]

I run it normally with --cargo-profile scripts/ci/gitlab/nextest.toml.

Um, you seem to be confused about which flag you passed. In logs in your other comments, I did not see any case where you passed --cargo-profile scripts/ci/gitlab/nextest.toml to cargo-llvm-cov.

(After unset NEXTEST_PROFILE,) try passing the exact same thing you passed to cargo-nextest to cargo-llvm-cov. https://github.com/taiki-e/cargo-llvm-cov/issues/267#issuecomment-1527674997 's patch should be able to handle it.

Maybe the problem is that you're passing the profile and not cargo-profile here

That should work correctly since parse_opt_passthrough refers to the current flag, not the variable name passed. In fact, https://github.com/taiki-e/cargo-llvm-cov/issues/267#issuecomment-1527714964 's log shows that it takes --cargo-profile testnet as input and runs --cargo-profile testnet:

--workspace --cargo-profile testnet --locked

[taiki-e]

I have confirmed that the https://github.com/taiki-e/cargo-llvm-cov/issues/267#issuecomment-1527674997 's patch locally can handle both options. So opened https://github.com/taiki-e/cargo-llvm-cov/pull/269.

Well, the real complication here is that it was --cargo-profile that was not being processed correctly prior to the https://github.com/taiki-e/cargo-llvm-cov/issues/267#issuecomment-1527674997 's patch, and in fact --profile is being processed correctly with or without the https://github.com/taiki-e/cargo-llvm-cov/issues/267#issuecomment-1527674997 's patch.

[taiki-e]

Fixed in v0.5.19

[TriplEight]

Sorry there was a typo:

No, with nextest I run it normally with --cargo-profile scripts/ci/gitlab/nextest.toml. (Maybe the problem is that you're passing the profile and not cargo-profile here)

I run it normally with --profile scripts/ci/gitlab/nextest.toml --cargo-profile testnet

[TriplEight]

Tested, and can't confirm it's working:

Installed package `cargo-llvm-cov v0.5.19` (executable `cargo-llvm-cov`)
$ cargo llvm-cov nextest --workspace --cargo-profile testnet --profile scripts/ci/gitlab/nextest.toml --verbose --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always --verbose --include-build-script --html --output-dir $ARCHIVE
Running `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --cargo-profile testnet --profile scripts/ci/gitlab/nextest.toml --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v`
error: profile `scripts/ci/gitlab/nextest.toml not found (known profiles: default, default-miri)`
error: process didn't exit successfully: `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --cargo-profile testnet --profile scripts/ci/gitlab/nextest.toml --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v` (exit status: 96)

[taiki-e]

Hmm, path-based profiles seem to be causing the problem? (I have read the documentation, and it is not even clear that such a profile is supported in the first place.)

cc @sunshowers

[TriplEight]

You are correct, in https://nexte.st/book/running.html there's

--config-file <PATH>
          Config file [default: workspace-root/.config/nextest.toml]

And --profile reads configured profiles from https://nexte.st/book/configuration.html. Trying to run with --config-file scripts/ci/gitlab/nextest.toml

[TriplEight]

Now it's working with

$ cargo llvm-cov nextest --workspace --cargo-profile testnet --config-file scripts/ci/gitlab/nextest.toml --verbose --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always --verbose --include-build-script --html --output-dir $ARCHIVE
[2023-04-29 12:55:12]      Running `CARGO_INCREMENTAL=0 CARGO_LLVM_COV=1 LLVM_PROFILE_FILE='/builds/TriplEight/target/llvm-cov-target/test-%p-%m.profraw' NEXTEST_TEST_THREADS=1 RUSTFLAGS='-Cdebug-assertions=y -Dwarnings -C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo nextest run --manifest-path /builds/TriplEight/Cargo.toml --target-dir /builds/TriplEight/target/llvm-cov-target --workspace --cargo-profile testnet --config-file scripts/ci/gitlab/nextest.toml --locked --features=runtime-benchmarks,runtime-metrics --no-fail-fast --color always -v`

There are no mentions of config-file in cargo-llvm-cov code. Checking if it's being passed to nextest.

UPD: can confirm --cargo-profile testnet is working as expected. Checking if --profile supports custom profile from --config-file scripts/ci/gitlab/nextest.toml

[taiki-e]

There are no mentions of config-file in cargo-llvm-cov code.

cargo-llvm-cov doesn't parse all the cargo-nextest CLI flags, but only those necessary for cargo-llvm-cov to work properly, and just passthrough everything else to the cargo-nextest.

This design eliminates the need to respond to each addition or deletion of CLI flags and also makes it easier to maintain compatibility with older cargo-nextest.

Checking if it's being passed to nextest.

It is passed to cargo-nextest. See this part in the log:

--cargo-profile testnet --config-file scripts/ci/gitlab/nextest.toml --verbose

[taiki-e]

Closing as it has been confirmed that passing the correct option (--config-file) works.