search

sourcefrog / cargo-mutants

:zombie: Inject bugs and see if your tests catch them!
https://mutants.rs/
MIT License
489 stars 27 forks source link

Baseline tests fail in nextest tree #226

Open sourcefrog opened 7 months ago

sourcefrog commented 7 months ago 
        FAIL [   1.824s] integration-tests::integration test_list_default

--- STDOUT:              integration-tests::integration test_list_default ---

running 1 test
test test_list_default ... FAILED

failures:

failures:
    test_list_default

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 14 filtered out; finished in 1.82s

--- STDERR:              integration-tests::integration test_list_default ---
thread 'test_list_default' panicked at integration-tests/tests/integration/fixtures.rs:201:13:
command failed:

command: "/ram/cargo-mutants-nextest-vxXZEX.tmp/target/debug/cargo-nextest-dup" "nextest" "--manifest-path" "/ram/nextest-fixtureUEjaIQ/src/Cargo.toml" "list" "--workspace" "--all-targets" "--message-format" "json"
exit code: Some(101)
--- stdout ---

--- stderr ---
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
info: experimental features enabled: setup-scripts
warning: /ram/nextest-fixtureUEjaIQ/src/Cargo.toml: only one of `license` or `license-file` is necessary
`license` should be used if the package license can be expressed with a standard SPDX expression.
`license-file` should be used if the package uses a non-standard license.
See https://doc.rust-lang.org/cargo/reference/manifest.html#the-license-and-license-file-fields for more information.
   Compiling uuid v1.2.1
   Compiling dylib-test v0.1.0 (/ram/nextest-fixtureUEjaIQ/src/dylib-test)
   Compiling cdylib-link v0.1.0 (/ram/nextest-fixtureUEjaIQ/src/cdylib/cdylib-link)
   Compiling with-build-script v0.1.0 (/ram/nextest-fixtureUEjaIQ/src/with-build-script)
   Compiling cdylib-example v0.1.0 (/ram/nextest-fixtureUEjaIQ/src/cdylib/cdylib-example)
   Compiling nextest-derive v0.1.0 (/ram/nextest-fixtureUEjaIQ/src/derive)
   Compiling nextest-tests v0.1.0 (/ram/nextest-fixtureUEjaIQ/src)
error: failed to run custom build command for `cdylib-link v0.1.0 (/ram/nextest-fixtureUEjaIQ/src/cdylib/cdylib-link)`

Caused by:
  process didn't exit successfully: `/ram/nextest-fixtureUEjaIQ/src/target/debug/build/cdylib-link-e65970b22ab3b7db/build-script-build` (exit status: 101)
  --- stderr
  warning: /ram/nextest-fixtureUEjaIQ/src/Cargo.toml: only one of `license` or `license-file` is necessary
  `license` should be used if the package license can be expressed with a standard SPDX expression.
  `license-file` should be used if the package uses a non-standard license.
  See https://doc.rust-lang.org/cargo/reference/manifest.html#the-license-and-license-file-fields for more information.
  error: failed to get `uuid` as a dependency of package `nextest-tests v0.1.0 (/ram/nextest-fixtureUEjaIQ/src)`

  Caused by:
    failed to load source for dependency `uuid`

  Caused by:
    Unable to update registry `crates-io`

  Caused by:
    failed to update replaced source registry `crates-io`

  Caused by:
    failed to read root of directory source: /ram/nextest-fixtureUEjaIQ/src/vendor

  Caused by:
    No such file or directory (os error 2)
  thread 'main' panicked at cdylib/cdylib-link/build.rs:23:9:
  cargo build failed with status Some(101)
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: cannot satisfy dependencies so `std` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `core` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `compiler_builtins` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `rustc_std_workspace_core` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `alloc` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `libc` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `unwind` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `cfg_if` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `miniz_oxide` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `adler` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `hashbrown` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `rustc_std_workspace_alloc` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `std_detect` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `rustc_demangle` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `addr2line` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `gimli` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `object` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `memchr` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `panic_unwind` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: could not compile `nextest-tests` (lib test) due to 19 previous errors
error: command `/home/mbp/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo test --no-run --message-format json-render-diagnostics --workspace --all-targets` exited with code 101

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        PASS [   1.998s] integration-tests::integration test_setup_script_error
------------
     Summary [   2.000s] 27/215 tests run: 6 passed, 21 failed, 0 skipped
        FAIL [   1.769s] integration-tests::integration test_list_binaries_only
        FAIL [   1.824s] integration-tests::integration test_list_default
        FAIL [   1.796s] integration-tests::integration test_list_full
        FAIL [   1.734s] integration-tests::integration test_list_full_after_build
        FAIL [   1.803s] integration-tests::integration test_list_host_after_build
        FAIL [   1.808s] integration-tests::integration test_list_target_after_build
        FAIL [   1.715s] integration-tests::integration test_relocated_run
        FAIL [   1.777s] integration-tests::integration test_run
        FAIL [   1.820s] integration-tests::integration test_run_after_build
        FAIL [   1.741s] integration-tests::integration test_run_from_archive
        FAIL [   1.777s] integration-tests::integration test_show_config_test_groups
        FAIL [   0.064s] integration-tests::integration test_target_dir
        FAIL [   0.080s]    nextest-runner::integration basic::test_filter_expr_with_string_filters
        FAIL [   0.074s]    nextest-runner::integration basic::test_filter_expr_without_string_filters
        FAIL [   0.069s]    nextest-runner::integration basic::test_list_binaries
        FAIL [   0.077s]    nextest-runner::integration basic::test_list_tests
        FAIL [   0.080s]    nextest-runner::integration basic::test_retries::retry_overrides_ignored
        FAIL [   0.073s]    nextest-runner::integration basic::test_retries::retry_overrides_obeyed
        FAIL [   0.073s]    nextest-runner::integration basic::test_run
        FAIL [   0.076s]    nextest-runner::integration basic::test_run_ignored
        FAIL [   0.069s]    nextest-runner::integration basic::test_string_filters_without_filter_expr
error: test run failed

*** result: Failure

  47.417917145s ERROR cargo test failed in an unmutated tree, so no mutants were tested

Using the in-progress support for running in-place (#157), I can reproduce this running in-place, so it's not a problem with copying.

The log shows that cargo-mutants runs

/home/mbp/.rustup/toolchains/beta-x86_64-unknown-linux-gnu/bin/cargo nextest run --package cargo-nextest --package nextest-filtering --package nextest-metadata --package nextest-runner --package quick-junit

and that does fail in a similar way in this tree:

     Summary [   1.168s] 160/198 tests run: 149 passed, 11 failed, 0 skipped
        FAIL [   1.050s] nextest-runner::integration basic::test_filter_expr_with_string_filters
        FAIL [   1.077s] nextest-runner::integration basic::test_filter_expr_without_string_filters
        FAIL [   1.097s] nextest-runner::integration basic::test_list_tests
        FAIL [   1.056s] nextest-runner::integration basic::test_retries::retry_overrides_ignored
        FAIL [   1.018s] nextest-runner::integration basic::test_retries::retry_overrides_obeyed
        FAIL [   1.096s] nextest-runner::integration basic::test_run
        FAIL [   1.103s] nextest-runner::integration basic::test_run_ignored
        FAIL [   1.079s] nextest-runner::integration basic::test_string_filters_without_filter_expr
        FAIL [   1.094s] nextest-runner::integration basic::test_termination
        FAIL [   1.045s] nextest-runner::integration target_runner::test_listing_with_target_runner
        FAIL [   1.056s] nextest-runner::integration target_runner::test_run_with_target_runner
error: test run failed

Very interestingly, invoking it with cargo +beta and otherwise the same args does pass, and also without the +beta. It also passes with the explicit path to e.g. ~/.cargo/bin/cargo.

But, it fails if I give it the path to a toolchain-specific binary, like ~/.rustup/toolchains/beta-x86_64-unknown-linux-gnu/bin/cargo.

I guess rustup installs a wrapper binary in ~/.cargo/bin/cargo (??) and perhaps it's setting an environment variable or otherwise doing something that nextest depends upon.

The nextest errors have a badly printed Vec<u8> like this:

thread 'basic::test_run_ignored' panicked at nextest-runner/tests/integration/fixtures.rs:382:10:
test list successfully created: CommandFail { binary_id: RustBinaryId("cdylib-example"), command: ["/home/mbp/src/nextest/fixtures/nextest-tests/target/debug/deps/cdylib_example-9222c584dde9dbd0", "--list", "--format", "terse"], exit_status: ExitStatus(unix_wait_status(32512)), stdout: [], stderr: [47, 104, 111, 109, 101, 47, 109, 98, 112, 47, 115, 114, 99, 47, 110, 101, 120, 116, 101, 115, 116, 47, 102, 105, 120, 116, 117, 114, 101, 115, 47, 110, 101, 120, 116, 101, 115, 116, 45, 116, 101, 115, 116, 115, 47, 116, 97, 114, 103, 101, 116, 47, 100, 101, 98, 117, 103, 47, 100, 101, 112, 115, 47, 99, 100, 121, 108, 105, 98, 95, 101, 120, 97, 109, 112, 108, 101, 45, 57, 50, 50, 50, 99, 53, 56, 52, 100, 100, 101, 57, 100, 98, 100, 48, 58, 32, 101, 114, 114, 111, 114, 32, 119, 104, 105, 108, 101, 32, 108, 111, 97, 100, 105, 110, 103, 32, 115, 104, 97, 114, 101, 100, 32, 108, 105, 98, 114, 97, 114, 105, 101, 115, 58, 32, 108, 105, 98, 116, 101, 115, 116, 45, 99, 98, 97, 98, 55, 49, 54, 57, 55, 100, 102, 57, 55, 98, 48, 48, 46, 115, 111, 58, 32, 99, 97, 110, 110, 111, 116, 32, 111, 112, 101, 110, 32, 115, 104, 97, 114, 101, 100, 32, 111, 98, 106, 101, 99, 116, 32, 102, 105, 108, 101, 58, 32, 78, 111, 32, 115, 117, 99, 104, 32, 102, 105, 108, 101, 32, 111, 114, 32, 100, 105, 114, 101, 99, 116, 111, 114, 121, 10] }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

But after decoding that I can see the errors are really like this

--- STDOUT:              nextest-runner::integration basic::test_run ---

running 1 test
for `cdylib-example`, command `/home/mbp/src/nextest/fixtures/nextest-tests/target/debug/deps/cdylib_example-9222c584dde9dbd0 --list --format terse` exited with code 127
--- stdout:

--- stderr:
/home/mbp/src/nextest/fixtures/nextest-tests/target/debug/deps/cdylib_example-9222c584dde9dbd0: error while loading shared libraries: libtest-cbab71697df97b00.so: cannot open shared object file: No such file or directory

---
test basic::test_run ... FAILED

failures:

failures:
    basic::test_run

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 23 filtered out; finished in 0.81s

--- STDERR:              nextest-runner::integration basic::test_run ---
warning: /home/mbp/src/nextest/fixtures/nextest-tests/Cargo.toml: only one of `license` or `license-file` is necessary
`license` should be used if the package license can be expressed with a standard SPDX expression.
`license-file` should be used if the package uses a non-standard license.
See https://doc.rust-lang.org/cargo/reference/manifest.html#the-license-and-license-file-fields for more information.
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on build directory
    Finished test [unoptimized + debuginfo] target(s) in 0.70s
thread 'basic::test_run' panicked at nextest-runner/tests/integration/fixtures.rs:387:17:
failed to create test list
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I can reproduce the same failure if I run the fixture binary directly, outside of the tests:

; ./fixtures/nextest-tests/target/debug/examples/other-2024fe2de6ada34d
./fixtures/nextest-tests/target/debug/examples/other-2024fe2de6ada34d: error while loading shared libraries: libtest-cbab71697df97b00.so: cannot open shared object file: No such file or directory
sourcefrog commented 1 month ago

It looks like it's still happening, from a similar command

/home/mbp/.rustup/toolchains/beta-x86_64-unknown-linux-gnu/bin/cargo nextest run --package cargo-nextest --package integration-tests --package nextest-filtering --package nextest-metadata --package nextest-runner

However strangely when I run that outside cargo-mutant it seems to pass...

sourcefrog commented 1 month ago

I'm going to unassign and deprioritize this for now because while it might be pointing to a bug in cargo-mutants or a failure to handle some reasonable behavior, it also seems really entangled with some quirks in Nextest.