sourcefrog / cargo-mutants

:zombie: Inject bugs and see if your tests catch them!
https://mutants.rs/
MIT License
558 stars 27 forks source link

Notice test failures that accompany hangs #423

Open kpreid opened 4 hours ago

kpreid commented 4 hours ago

Consider a case where a mutation causes a hang, but is also detected by a test:

pub struct Iter(u32);
impl Iterator for Iter {
    type Item = u32;
    fn next(&mut self) -> Option<u32> {
        if self.0 == 0 {
            None
        } else {
            self.0 -= 1;
            Some(self.0)
        }
    }
}

#[test]
fn t1() {
    assert_eq!(Iter(10).next(), Some(9));
}

#[test]
fn t2() {
    for _ in Iter(10) {}
}

In this case, mutations of Iter::next() may cause t1 to fail and t2 to hang. This is counted as a timeout, but could be detected rapidly as a caught mutant, and this might make cargo-mutants more useful on codebases that have a lot of this kind of code (stopping conditions that can be trivially mutated into running forever).

Detecting this case would require parsing the test harness output which could sometimes be corrupted (stray logging to real stderr) or arbitrary (custom test harnesses), so it should be optional.

(In the example case above, it would arguably be wise for t2 to have its own maximum-iterations guard condition, but in the general case, the loop may be part of code that is not test code and cannot predict the maximum number of iterations.)

sourcefrog commented 3 hours ago

Yep, https://github.com/sourcefrog/cargo-mutants/issues/194 has some other related ideas. I think the heart of it is: the standard Rust test framework does not exit early when one test fails, but at least for the cargo mutants use case it would be better if it did.