Integration tests failures in the last release

orhun commented 1 week ago

What happened?

I was packaging the latest available version of cargo-insta (1.40.0) for Arch Linux and hit the following test failures in a clean chroot:

Details

``` running 11 tests test test_root_crate_no_all ... FAILED test test_virtual_manifest_all ... FAILED test test_virtual_manifest_single_crate ... FAILED test test_virtual_manifest_default ... FAILED test test_root_crate_all ... FAILED test test_utf8_inline ... FAILED test test_yaml_inline ... FAILED test test_hashtag_escape_in_inline_snapshot ... FAILED test test_json_inline ... FAILED test test_force_update_snapshots ... FAILED test test_force_update_inline_snapshot ... FAILED failures: ---- test_root_crate_no_all stdout ---- error: `cargo metadata` exited with an error: error: failed to load manifest for workspace member `/tmp/.tmpwPqEX7/member` referenced by workspace at `/tmp/.tmpwPqEX7/Cargo.toml` Caused by: failed to load manifest for dependency `insta` Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) thread 'test_root_crate_no_all' panicked at tests/main.rs:76:5: Tests failed: error: `cargo metadata` exited with an error: error: failed to load manifest for workspace member `/tmp/.tmpwPqEX7/member` referenced by workspace at `/tmp/.tmpwPqEX7/Cargo.toml` Caused by: failed to load manifest for dependency `insta` Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ---- test_virtual_manifest_all stdout ---- error: `cargo metadata` exited with an error: error: failed to load manifest for workspace member `/tmp/.tmpQcCB7N/member-1` referenced by workspace at `/tmp/.tmpQcCB7N/Cargo.toml` Caused by: failed to load manifest for dependency `insta` Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) thread 'test_virtual_manifest_all' panicked at tests/main.rs:76:5: Tests failed: error: `cargo metadata` exited with an error: error: failed to load manifest for workspace member `/tmp/.tmpQcCB7N/member-1` referenced by workspace at `/tmp/.tmpQcCB7N/Cargo.toml` Caused by: failed to load manifest for dependency `insta` Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) ---- test_virtual_manifest_single_crate stdout ---- error: `cargo metadata` exited with an error: error: failed to load manifest for workspace member `/tmp/.tmpGsHa9d/member-1` referenced by workspace at `/tmp/.tmpGsHa9d/Cargo.toml` Caused by: failed to load manifest for dependency `insta` Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) thread 'test_virtual_manifest_single_crate' panicked at tests/main.rs:76:5: Tests failed: error: `cargo metadata` exited with an error: error: failed to load manifest for workspace member `/tmp/.tmpGsHa9d/member-1` referenced by workspace at `/tmp/.tmpGsHa9d/Cargo.toml` Caused by: failed to load manifest for dependency `insta` Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) ---- test_virtual_manifest_default stdout ---- error: `cargo metadata` exited with an error: error: failed to load manifest for workspace member `/tmp/.tmpNXpSwH/member-1` referenced by workspace at `/tmp/.tmpNXpSwH/Cargo.toml` Caused by: failed to load manifest for dependency `insta` Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) thread 'test_virtual_manifest_default' panicked at tests/main.rs:76:5: Tests failed: error: `cargo metadata` exited with an error: error: failed to load manifest for workspace member `/tmp/.tmpNXpSwH/member-1` referenced by workspace at `/tmp/.tmpNXpSwH/Cargo.toml` Caused by: failed to load manifest for dependency `insta` Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) ---- test_root_crate_all stdout ---- error: `cargo metadata` exited with an error: error: failed to load manifest for workspace member `/tmp/.tmpSSsyBN/member` referenced by workspace at `/tmp/.tmpSSsyBN/Cargo.toml` Caused by: failed to load manifest for dependency `insta` Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) thread 'test_root_crate_all' panicked at tests/main.rs:76:5: Tests failed: error: `cargo metadata` exited with an error: error: failed to load manifest for workspace member `/tmp/.tmpSSsyBN/member` referenced by workspace at `/tmp/.tmpSSsyBN/Cargo.toml` Caused by: failed to load manifest for dependency `insta` Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) ---- test_utf8_inline stdout ---- error: failed to get `insta` as a dependency of package `test_utf8_inline v0.1.0 (/tmp/.tmpS0wjnU)` Caused by: failed to load source for dependency `insta` Caused by: Unable to update /build/cargo-insta/src/insta Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) done: no snapshots to review thread 'test_utf8_inline' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review error: failed to get `insta` as a dependency of package `test_utf8_inline v0.1.0 (/tmp/.tmpS0wjnU)` Caused by: failed to load source for dependency `insta` Caused by: Unable to update /build/cargo-insta/src/insta Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) ---- test_yaml_inline stdout ---- error: failed to get `insta` as a dependency of package `test_yaml_inline v0.1.0 (/tmp/.tmpZPbHXs)` Caused by: failed to load source for dependency `insta` Caused by: Unable to update /build/cargo-insta/src/insta Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) done: no snapshots to review thread 'test_yaml_inline' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review error: failed to get `insta` as a dependency of package `test_yaml_inline v0.1.0 (/tmp/.tmpZPbHXs)` Caused by: failed to load source for dependency `insta` Caused by: Unable to update /build/cargo-insta/src/insta Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) ---- test_hashtag_escape_in_inline_snapshot stdout ---- error: failed to get `insta` as a dependency of package `test_hashtag_escape v0.1.0 (/tmp/.tmpXnQRHR)` Caused by: failed to load source for dependency `insta` Caused by: Unable to update /build/cargo-insta/src/insta Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) done: no snapshots to review thread 'test_hashtag_escape_in_inline_snapshot' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review error: failed to get `insta` as a dependency of package `test_hashtag_escape v0.1.0 (/tmp/.tmpXnQRHR)` Caused by: failed to load source for dependency `insta` Caused by: Unable to update /build/cargo-insta/src/insta Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) ---- test_json_inline stdout ---- error: failed to get `insta` as a dependency of package `test_json_inline v0.1.0 (/tmp/.tmp0u7u6d)` Caused by: failed to load source for dependency `insta` Caused by: Unable to update /build/cargo-insta/src/insta Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) done: no snapshots to review thread 'test_json_inline' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review error: failed to get `insta` as a dependency of package `test_json_inline v0.1.0 (/tmp/.tmp0u7u6d)` Caused by: failed to load source for dependency `insta` Caused by: Unable to update /build/cargo-insta/src/insta Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) ---- test_force_update_snapshots stdout ---- error: failed to get `insta` as a dependency of package `test_force_update_current v0.1.0 (/tmp/.tmpkbv81m)` Caused by: failed to load source for dependency `insta` Caused by: Unable to update /build/cargo-insta/src/insta Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) done: no snapshots to review thread 'test_force_update_snapshots' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review error: failed to get `insta` as a dependency of package `test_force_update_current v0.1.0 (/tmp/.tmpkbv81m)` Caused by: failed to load source for dependency `insta` Caused by: Unable to update /build/cargo-insta/src/insta Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) ---- test_force_update_inline_snapshot stdout ---- error: failed to get `insta` as a dependency of package `force-update-inline v0.1.0 (/tmp/.tmpjcknO2)` Caused by: failed to load source for dependency `insta` Caused by: Unable to update /build/cargo-insta/src/insta Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) done: no snapshots to review thread 'test_force_update_inline_snapshot' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review error: failed to get `insta` as a dependency of package `force-update-inline v0.1.0 (/tmp/.tmpjcknO2)` Caused by: failed to load source for dependency `insta` Caused by: Unable to update /build/cargo-insta/src/insta Caused by: failed to read `/build/cargo-insta/src/insta/Cargo.toml` Caused by: No such file or directory (os error 2) failures: test_force_update_inline_snapshot test_force_update_snapshots test_hashtag_escape_in_inline_snapshot test_json_inline test_root_crate_all test_root_crate_no_all test_utf8_inline test_virtual_manifest_all test_virtual_manifest_default test_virtual_manifest_single_crate test_yaml_inline test result: FAILED. 0 passed; 11 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s error: test failed, to rerun pass `--test main` ```

Upon looking closer, I realized cargo test was looking for a directory named insta instead of cargo-insta-1.40.0 so I updated the packaging instructions. Now hitting a different failure:

Details

``` running 11 tests test test_virtual_manifest_single_crate ... FAILED test test_force_update_inline_snapshot ... FAILED test test_utf8_inline ... FAILED test test_hashtag_escape_in_inline_snapshot ... FAILED test test_yaml_inline ... FAILED test test_virtual_manifest_default ... FAILED test test_force_update_snapshots ... FAILED test test_root_crate_no_all ... FAILED test test_virtual_manifest_all ... FAILED test test_root_crate_all ... FAILED test test_json_inline ... FAILED failures: ---- test_virtual_manifest_single_crate stdout ---- warning: virtual workspace defaulting to `resolver = "1"` despite one or more workspace members being on edition 2021 which implies `resolver = "2"` note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest note: for more details see https://doc.rust-lang.org/cargo/reference/resolver.html#resolver-versions error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `virtual-manifest-single-member-1 v0.1.0 (/tmp/.tmpQIRv6U/member-1)` done: no snapshots to review thread 'test_virtual_manifest_single_crate' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review warning: virtual workspace defaulting to `resolver = "1"` despite one or more workspace members being on edition 2021 which implies `resolver = "2"` note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest note: for more details see https://doc.rust-lang.org/cargo/reference/resolver.html#resolver-versions error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `virtual-manifest-single-member-1 v0.1.0 (/tmp/.tmpQIRv6U/member-1)` note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ---- test_force_update_inline_snapshot stdout ---- Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `force-update-inline v0.1.0 (/tmp/.tmpU1X4i0)` done: no snapshots to review thread 'test_force_update_inline_snapshot' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `force-update-inline v0.1.0 (/tmp/.tmpU1X4i0)` ---- test_utf8_inline stdout ---- Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `test_utf8_inline v0.1.0 (/tmp/.tmpzyxZKK)` done: no snapshots to review thread 'test_utf8_inline' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `test_utf8_inline v0.1.0 (/tmp/.tmpzyxZKK)` ---- test_hashtag_escape_in_inline_snapshot stdout ---- Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `test_hashtag_escape v0.1.0 (/tmp/.tmpJziAP1)` done: no snapshots to review thread 'test_hashtag_escape_in_inline_snapshot' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `test_hashtag_escape v0.1.0 (/tmp/.tmpJziAP1)` ---- test_yaml_inline stdout ---- Blocking waiting for file lock on package cache Updating crates.io index error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `test_yaml_inline v0.1.0 (/tmp/.tmp96XyRv)` done: no snapshots to review thread 'test_yaml_inline' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review Blocking waiting for file lock on package cache Updating crates.io index error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `test_yaml_inline v0.1.0 (/tmp/.tmp96XyRv)` ---- test_virtual_manifest_default stdout ---- warning: virtual workspace defaulting to `resolver = "1"` despite one or more workspace members being on edition 2021 which implies `resolver = "2"` note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest note: for more details see https://doc.rust-lang.org/cargo/reference/resolver.html#resolver-versions error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `virtual-manifest-default-member-1 v0.1.0 (/tmp/.tmpIbVyUt/member-1)` done: no snapshots to review thread 'test_virtual_manifest_default' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review warning: virtual workspace defaulting to `resolver = "1"` despite one or more workspace members being on edition 2021 which implies `resolver = "2"` note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest note: for more details see https://doc.rust-lang.org/cargo/reference/resolver.html#resolver-versions error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `virtual-manifest-default-member-1 v0.1.0 (/tmp/.tmpIbVyUt/member-1)` ---- test_force_update_snapshots stdout ---- Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `test_force_update_current v0.1.0 (/tmp/.tmpnkAQOC)` done: no snapshots to review thread 'test_force_update_snapshots' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `test_force_update_current v0.1.0 (/tmp/.tmpnkAQOC)` ---- test_root_crate_no_all stdout ---- Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `root-crate-no-all-member v0.0.0 (/tmp/.tmpHMnWL5/member)` done: no snapshots to review thread 'test_root_crate_no_all' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `root-crate-no-all-member v0.0.0 (/tmp/.tmpHMnWL5/member)` ---- test_virtual_manifest_all stdout ---- warning: virtual workspace defaulting to `resolver = "1"` despite one or more workspace members being on edition 2021 which implies `resolver = "2"` note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest note: for more details see https://doc.rust-lang.org/cargo/reference/resolver.html#resolver-versions Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `virtual-manifest-all-member-1 v0.1.0 (/tmp/.tmpYe0GGS/member-1)` done: no snapshots to review thread 'test_virtual_manifest_all' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review warning: virtual workspace defaulting to `resolver = "1"` despite one or more workspace members being on edition 2021 which implies `resolver = "2"` note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest note: for more details see https://doc.rust-lang.org/cargo/reference/resolver.html#resolver-versions Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `virtual-manifest-all-member-1 v0.1.0 (/tmp/.tmpYe0GGS/member-1)` ---- test_root_crate_all stdout ---- Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `root-crate-all-member v0.0.0 (/tmp/.tmpXaLYOu/member)` done: no snapshots to review thread 'test_root_crate_all' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review Blocking waiting for file lock on package cache error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `root-crate-all-member v0.0.0 (/tmp/.tmpXaLYOu/member)` ---- test_json_inline stdout ---- Updating crates.io index error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `test_json_inline v0.1.0 (/tmp/.tmppu1Pcz)` done: no snapshots to review thread 'test_json_inline' panicked at tests/main.rs:76:5: Tests failed: done: no snapshots to review Updating crates.io index error: no matching package named `insta` found location searched: /build/cargo-insta/src/insta required by package `test_json_inline v0.1.0 (/tmp/.tmppu1Pcz)` failures: test_force_update_inline_snapshot test_force_update_snapshots test_hashtag_escape_in_inline_snapshot test_json_inline test_root_crate_all test_root_crate_no_all test_utf8_inline test_virtual_manifest_all test_virtual_manifest_default test_virtual_manifest_single_crate test_yaml_inline test result: FAILED. 0 passed; 11 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s error: test failed, to rerun pass `--test main` ```

Reproduction steps

Fetch the PKGBUILD from https://gitlab.archlinux.org/archlinux/packaging/packages/cargo-insta
Update pkgver and checksums accordingly to 1.40.0
Build in a clean chroot (extra-x86_64-build)

Insta Version

1.40.0

rustc Version

cargo 1.80.1 (376290515 2024-07-16)

What did you expect?

Successful build.

orhun commented 1 week ago

I disabled the tests and pushed the package for now. It would be nice to enable the tests though :)

max-sixty commented 1 week ago

Thanks @orhun and sorry for the issue.

Do you have any initial take on why the arch build environment would be different? We do run these tests in CI (& developer machines) and they pass. Are they by any chance renaming the path within the repo?

The tests are new to this release and are much broader than those we've had before — they each run the binary on a real project they create in a temp path — so not shocked that they're a bit more susceptible to env differences.

max-sixty commented 1 week ago

I had a look at the commands in https://gitlab.archlinux.org/archlinux/packaging/packages/cargo-insta/-/blob/main/PKGBUILD?ref_type=heads — I can't reproduce the errors. Do you have a set of commands that I can run to reproduce?

orhun commented 1 week ago

Yup, you need to build the package in a clean chroot. If you are already using Arch Linux, running extra-x86_64-build command would be enough :)

Do you have any initial take on why the arch build environment would be different? We do run these tests in CI (& developer machines) and they pass. Are they by any chance renaming the path within the repo?

The repository path is cloned as $pkgname-$pkgver (see the first logs above), but then I renamed it to insta and got another error (see the second error).

I don't think we modify anything else in the repo.

The tests are new to this release and are much broader than those we've had before — they each run the binary on a real project they create in a temp path — so not shocked that they're a bit more susceptible to env differences.

I'm not sure how the tests are being run right now, maybe it would be better if there was an example code / documentation to look at to figure things out further. Do you do anything special while running these tests in CI?

max-sixty commented 1 week ago

I'm not sure how the tests are being run right now, maybe it would be better if there was an example code / documentation to look at to figure things out further. Do you do anything special while running these tests in CI?

Documented here: https://github.com/mitsuhiko/insta/blob/ab73548561755f79b430f088df9b96fd014e161a/.github/workflows/tests.yml#L9-L27

This runs make test but also just cargo test will work

In the PKGBUILD file, what's the URL that it's downloading from? I get a 404 from https://github.com/mitsuhiko/insta/archive/v1.40.0.tar.gz; I'm probably making a basic mistake...

orhun commented 6 days ago

In the PKGBUILD file, what's the URL that it's downloading from? I get a 404 from v1.40.0.tar.gz; I'm probably making a basic mistake...

Nope, you were right.

https://gitlab.archlinux.org/archlinux/packaging/packages/cargo-insta/-/issues/1(Source does not exist)

The issue is fixed after updating the source of the package. I guess we were building from leftover artifacts on the build server - or something like that.

That's a bit weird, but hey, everything works now.

Sorry for wasting your time here a bit :/

max-sixty commented 6 days ago

No stress, thanks for debugging!

And for the packaging!

mitsuhiko / insta