search

tokio-rs / loom

Concurrency permutation testing tool for Rust.
MIT License
2.08k stars 110 forks source link

per-test checkpoint files #271

Open hawkw opened 2 years ago

hawkw commented 2 years ago

Currently, when running with checkpointing enabled, loom will generate a single checkpoint file. When a test fails, cargo test will continue running tests, and may overwrite that checkpoint file. This means that checkpointing can really only be used when running one test at a time.

This is unfortunate, as it prevents workflows where all the tests are run with checkpoints enabled to generate a checkpoint file for each failing test, which can then be re-run.

We should find some way to generate separate checkpoint files for each test, so that running additional tests doesn't clobber a checkpoint file.

hawkw commented 2 years ago

The tricky part here is generating a unique, durable identifier for the current test.

If cargo test is not run with --test-threads=1, the current thread name should be the name of the test. But, this won't work with --test-threads=1. Another approach might be to have loom::model and loom::model::Buiilder::check have #[track_caller] attributes and capture the source location where check is called from. But this falls down in a case where the user has their own function that wraps Builder::check or model that doesn't also have a #[track_caller] on it.

If we could ensure that loom controls the invocation of these functions, such as by having a special #[loom::test] proc macro, we could ensure that those locations are correct. This is what the proptest crate does for its regression checkpointing system. But, currently, loom works nicely with the standard #[test] attribute, and it would be a shame to give that up...