Running a test archive without copy of source

[denniskempin]

Hi nextest team!

I would like to replace the custom test runner we built for our project with nextest, since it fixes many of the same issues we had with plain cargo test and does it a lot better than our python script.

While nextest supports packaging tests in an archive to run on another device, we execute tests on special devices that do not have the source code available, which is currently required by nextest to run an archive.

Is this is a use-case that could be supported by nextest with some modifications? I would be happy to contribute code if you can give me some pointers.

Alternatively, could you advise on how this could be achieved with a process on top of nextest? A bit of experimenting shows that nextest run will work if only the *.toml and rust-toolchain are available on the target device. However I'd rather build on a more solid foundation than that.

[sunshowers]

Hi there -- thanks for your interest!

The reasons nextest requires source code while running builds are that:

• sometimes, tests access files in the repo (e.g. test fixtures)
• we need to set the cwds properly for each test

This hasn't been a burden for most people, but in a device testing scenario I can definitely see it being a hassle.

In your case, it sounds that your test doesn't need access to any source files. In that case, all that would be required is:

1. .config/nextest.toml and .cargo/config[.toml] files stored inside the repo, if you care about them
2. the directory structure of the repo, specifically the directories where all the crate Cargo.tomls live -- if you're executing cargo-nextest directly as cargo-nextest nextest run --archive ..., then if I remember correctly not even the Cargo.toml or rust-toolchain files should be required.

Regarding stability: I don't expect this list to change in the short to medium term. Archiving and reusing builds hasn't been touched in a while, and the main missing thing (creating partitioned archives) is not going to impact this.

Making nextest automatically manage this: It would definitely be interesting to add a "skeleton mode" where nextest automatically does this (or most of this) for you. But I don't have a good sense of the scenarios involved here, and it would be great if you first tried out the above scheme (should hopefully be very little work) and we got some real-world feedback.

In particular, I'm worried that not having access to the source code for test fixtures would be surprising to developers. In your case they're already likely dealing with this, but others interested in this feature might not be. We'd have to add documentation and messaging around this to make it clear that if users try out this "skeleton mode" and it doesn't work, it likely isn't nextest that's at issue.

[sunshowers]

BTW if you have an external issue tracker I'd definitely be interested in subscribing to it!

[denniskempin]

Thank you very much for the info, this is really helpful!

I will prototype a new version of our test runner using nextest and see if I get it to work by bundling the right files / directory structure. I filed an issue here that you can follow: https://issuetracker.google.com/issues/261600801

One problem I ran into when not all files are present is that libtest-*.so cannot be found.

e.g. when run with just the directory structure and only the root Cargo.toml:

❯ cargo-nextest nextest run --archive-file /workspace/crosvm/a/testpkg.tar.zst --workspace-remap .
  Extracting 70 binaries (including 1 non-test binaries) and 7 linked paths to /tmp/nextest-archive-IjuTyx
   Extracted 848 files to /tmp/nextest-archive-IjuTyx in 3.56s
error: creating test list failed

Caused by:
  for `argh_helpers::proc-macro/argh_helpers`, command `/tmp/nextest-archive-IjuTyx/target/debug/deps/argh_helpers-2c5bd6db145e360c --list --format terse` exited with code 127
--- stdout:

--- stderr:
/tmp/nextest-archive-IjuTyx/target/debug/deps/argh_helpers-2c5bd6db145e360c: error while loading shared libraries: libtest-ef2aecd079814f3c.so: cannot open shared object file: No such file or directory

---

It'll only find that shared library once I have copied enough files for cargo to be able to at least generate metadata (i.e. including all Cargo.toml, lib.rs, build.rs, test/*.rs, etc).

I will go ahead with that bundle of files for now. Frankly, next to the 100s of MBs of test binaries, it's a drop in the bucket anyway.

Re automatically managing this in nextest:

I do find it quite intuitive that if running tests on another machine, local source files won't be available. But I am coming from a very different context where we frequently test on production hardware rather than CI builders, so my intuition is probably off.

Maybe a CLI option --no-workspace in place of --workspace-remap would be an intuitive way of handling this. Without a workspace, there are no source files available.

[sunshowers]

Oh that's really interesting re not being able to find the libtest.so files. Wonder if we should just bundle them with the archive.