search

sourcefrog / cargo-mutants

:zombie: Inject bugs and see if your tests catch them!
https://mutants.rs/
MIT License
543 stars 27 forks source link

Run all workspace tests #394

Open RossSmyth opened 1 month ago

RossSmyth commented 1 month ago

Hello, thanks for this neat project.

I decided to run this for my main project and saw almost every test fail, which was surprising to me. My analysis afterwards showed that it modifies a crate, then has cargo run the tests associated with that crate.

This is fine for some people, but for my project (and others!) this is not. I very specifically do not have almost any tests in the individual workspace crates. Instead, all tests are in a single crate. This is for two reason:

  1. Test speed

Previously my test suite took about 20-30 seconds to run. Not too bad, but when I'm doing quick iterations it's annoying to wait. After refactoring it running the whole test suite takes 0.5 seconds.

  1. API privacy

Tests should only test the public API. This forces that.

This was inspired by Rust-Analyzer's approach and works quite well in practice. The specific model I use is used in the project jj here, where there is one file you add test modules to. This means that the assumption stated in the book

For each mutant, only the containing package's tests are run, on the theory that each package's tests are responsible for testing the package's code.

It is definitely not true. You did mention here that running all tests is a future possibility, and I do think this would be valuable.

RossSmyth commented 1 month ago

And to be clear this is just a request for an option like --run-all or something.

sourcefrog commented 1 month ago

Hey, thanks for pointing out this use case in jj (which is a project I was excited to read about, too.)

I think something like --test-all-packages or --test-package=A,B,C would make sense to add, including in mutants.toml.

sourcefrog commented 1 week ago

cc @panarch