Detect timeouts in advance

[richardwerkman]

Is your feature request related to a problem? Please describe. Timeouts will take a long time to detect when tested.

Describe the solution you'd like We can detect obvious timeouts before even testing and mark them as timeout in advance. For example:

for (let i=0; i<something; i++)

Obviously this will result in a timeout when Stryker does its thing and mutates it to:

for (let i=0; i<something; i--)

We could detect this inside the mutator that places the mutation or detect them after placing.

Describe alternatives you've considered There might be more obvious timeout cases, but we have to look at existing reports to spot these.

Additional context This was originaly posted in our stryker slack chanel by Sander Van Beek

[dupdob]

I am afraid this is harder than it looks at first glance. You are dealing with the halting problem. E.G: there may be conditional exits if (...) break; somewhere in the loop, which means that may not be a time out after all....

[richardwerkman]

But is it worth to place them only for those few cases where it will not result in a timeout?

We place three mutations in a for loop:

Maybe we can skip one or two while still forcing tests?

[dupdob]

Makes sense. To be honest, I am still dubious about the interest of mutating the incrementor part of the for statement, mostly because we already mutate the condition part.

[richardwerkman]

Yes exactly. They might even be duplicate mutations. So I think we could just skip the incrementor part (which will save us from a timeout) and still force the same tests.

To me it sounds like a win-win 😄

[dupdob]

in line, and it requires less effort than attempting to identify timeouts

[dupdob]

quick question: should we:

1. avoid mutating expressions that would lead to timeouts
2. mark them as timeout on generation

I am more confortable with 1; I am afraid we could get wrong for two (with some non trivial constructions)