stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.75k stars 176 forks source link

Ability to disable mutant, that end up with timeout #2905

Closed CzBuCHi closed 3 weeks ago

CzBuCHi commented 2 months ago

Is your feature request related to a problem? Please describe.

I have this code:

List<string> list = new List<string>();
string Aggregate() { 
    var top = list[0];
    list.RemoveAt(0); 

    if (top.All(char.IsNumber)) {
        return top;
    }

    var left = Aggregate();
    var right = Aggregate();
    return "(" + left + top + right + ")";
}

when stryker removes list.RemoveAt(0); it will result in surviving mutant due to timeout ...

Describe the solution you'd like Ability to use 'timeout' keyword when disabling mutators via comments:

// Stryker disable once timeout
list.RemoveAt(0); 

Describe alternatives you've considered im currently using // Stryker disable once all but that feels wrong because some mutant may still squeeze in there ...

Additional context Same thing applies to uncovered parts of code:

public string Foo(int bar){
    if (bar == 3 || bar == 5) {
        return FooImpl(bar); // this is the only call to FooImpl
    }

    return "BAZ";
}

private string FooImpl(int bar) {
    return bar switch {
               3 => "FOO",
               5 => "BAR",
               // Stryker disable once all     <= this is a hole for mutant to sneak thru...
              // dotCover disable next line <= dotCover will not count next line to coverage
               _ => throw new UnreachableException(),
           };
}
rouke-broersma commented 2 months ago

We wouldn't know it's a timeout until after testing, so how can we disable the mutant from testing?

Instead of disable all you can specify the type of mutator to disable here.

CzBuCHi commented 2 months ago

i would interpret // Stryker disable once timeout as if timeout occurs after this line, do not show it in report

richardwerkman commented 2 months ago

it will result in surviving mutant due to timeout ...

This is not true, timeouts are counted as killed when calculating the mutation score. So why would you want to exclude timeouts from the report?

im currently using // Stryker disable once all but that feels wrong because some mutant may still squeeze in there ...

How would any mutant be generated here? Maybe I'm not following correctly.

dupdob commented 2 months ago

@CzBuCHi : what is exactly the problem you are trying to solve here?

CzBuCHi commented 2 months ago

probably my own confusion ( after reading @richardwerkman answer ) - i considered timeouts as bad & need to be fixed (its not green :/)

but i think this can still be applied to uncovered parts of code:

public string Foo(int bar){
    if (bar == 3 || bar == 5) {
        return FooImpl(bar); // this is the only call to FooImpl 
    }

    return "BAZ";
}
private string FooImpl(int bar) {
    return bar switch {
               3 => "FOO",
               5 => "BAR",
              // dotCover disable next line
               _ => throw new UnreachableException(), // this will be never called, but compiler requires it ...
           };
}

because in this case stryker will complain about uncovered parts of code ... (idk right now it that one is red or yellow)

dupdob commented 2 months ago

Stryker comments can be used for uncovered parts of the code, so this is already possible. The only feature request I can see here is recognizing/reusing existing comments (such as dotCover here)

rouke-broersma commented 3 weeks ago

We don't currently plan supporting comments from other frameworks