Cargo nextest integration

[somehowchris]

Describe the feature Simply said, I would like to use cargo nextest Instead of cargo test

Why? So I love cargo tarpaulin But for running test cargo nextest Is just simply much faster and has things like retires which makes work just more efficient.

Some Ressources:

• nextest homepage: https://nexte.st/book/installation.html
• Usage: https://nexte.st/book/usage.html
• Benchmark cargo test vs nextest: https://nexte.st/book/benchmarks.html
• LLVM test coverage: https://nexte.st/book/test-coverage.html

Now I haven’t dug into the cargo-tarpaulin Code as it doesn’t seem like a 30 min thing to do, so my general questions are. Would it be hard to integration something like this? Do you have any code parts or points to watch out if I would have a go on that?

As it seems —command Can be set to build Or test, how big would the effort be to have a third option?

[xd009642]

So I was going to suggest a vague idea of how to do it but as I think about it more I don't think it will work, with the current ptrace engine without a different flag added. The Command option does the build step as cargo build or cargo test --no-run, and those both output metadata on where the binaries end up (except for doc tests which are another edge case, they're detected in a folder from some other config opt to cargo). And then the binaries are ran directly - not sure if that works with nextest or if the nextest runner requires you to do cargo nextest :thinking:

Generally, with ptrace coverage tools like kcov or tarpaulin you need the binary to get coverage for before starting coverage collection to read the debug info, so building tests and running tests needs to be 2 separate stages with a stage between to identify coverage spots.

[xd009642]

I probably won't look at this soon as I'm rather overloaded but I can offer advice/guidance beyond this if you're will to contribute :sweat_smile: otherwise I may try and move some work projects to nextest and figure it out then a bit in future

[sunshowers]

Generally, with ptrace coverage tools like kcov or tarpaulin you need the binary to get coverage for before starting coverage collection to read the debug info, so building tests and running tests needs to be 2 separate stages with a stage between to identify coverage spots.

Thanks for looking into this! cargo nextest does support splitting up the build and run phases: https://nexte.st/book/reusing-builds.html

There's also support for listing tests with machine-readable output: https://nexte.st/book/machine-readable.html (in particular, --list-type binaries-only will produce just a JSON list of binaries which can be deserialized into https://docs.rs/nextest-metadata/latest/nextest_metadata/struct.BinaryListSummary.html).

Hope that helps!

[detly]

This also relates to something I mention in #1352: needing to run separate CI jobs for build + test, and for coverage. It's not possible to get jUnit XML output for the test results using libtest (any more), so in order to get jUnit reports for the test results I now run tests via cargo nextest. To get coverage I run cargo tarpaulin. The extra redundancy = longer time for CI runs.