add --fail-fast [feature]

[cehteh]

For a fast check it would be nice to have a --fail-fast flag. This should make cargo-mutants exit with a failure on the first 'MISSING' mutant. This should allow for much faster checks in a CI or pre-commit hook on a source tree thats formerly known to be succeed cargo mutants.

[sourcefrog]

I think this makes sense to have as an option and it would do no harm, although I do wonder a bit about the story for using it:

If we run it in CI then, it's true, we can reject the PR faster if we find one missed mutants. However, then presumably the author has to find and fix all the missed mutants, and so at some point it seems they need to run cargo-mutants again to find everything that's missed. This would mean re-checking many of the mutants that were already caught in CI. Perhaps the idea is that the developer re-runs mutants on their own machine rather than using CI capacity. But still, perhaps it would be faster overall if CI told them every missed mutants.

[cehteh]

Of course this is optional and opinionated. In some cases where you want to implement semi-automatic workflows in a CI you are only interested in if the software passes or fails some tests. Ideally at such a stage it should been already mutants checked earlier in the pipeline. We only want to make double sure it really is even in presence of inadequate pipeline specifications. Failing early would be less resource intensive and faster in this case.

[sourcefrog]

Should be pretty small if you want to try a PR: in the main loop in lab.rs bail out if we've seen a missed or timeout mutant.

[cehteh]

Should be pretty small if you want to try a PR: in the main loop in lab.rs bail out if we've seen a missed or timeout mutant.

I'll eventually look into that, may take some time. I think I wont bail on timeouts. Some mutants are safe and tested but introduce deadlocks/lifelocks as logic bugs that timeout. While this is unfortunate and should be addressed somehow, it is not a hard fail or missing test unless one would want to mandate a full deadlock detector just for these cases (unmutated code should be deadlock free), or require such cases to be skipped. I think either way is not a conclusive solution yet, better just leave it for now.