search

UT-SE-Research / iDFlakies

29 stars 33 forks source link

Added option to change round counting semantics for test run repetitions #8

Closed s-winter closed 5 years ago

s-winter commented 5 years ago

Added option to disable the 'round skipping' feature that aims for the execution of dt.randomize.rounds number of rounds w/o detection of new OD tests.

This can, for instance, be useful to ensure a maximum number of rounds to be executed for flaky test detection. To indicate that the number of rounds specified by dt.randomize.rounds is the maximum number of rounds, set the property dt.roundsemantics.total=true. If this is set, also rounds that do not find flakies are reported as individual log lines on the console.

winglam commented 5 years ago

So just to be clear, if dt.roundsemantics.total=true, then iDFlakies will execute exactly "rounds" number of rounds, correct?

Conversely, if dt.roundsemantics.total=false, then iDFlakies will execute as many rounds as needed until "rounds" number of rounds has not found any OD/NOD tests, correct?

s-winter commented 5 years ago

Correct, but I forgot to set a default value. So right now specifying the property is mandatory. Let me change that. I would set the default to false, as this resembles the original behavior.

s-winter commented 5 years ago

Updated and tested.

winglam commented 5 years ago

LGTM! Thanks for the changes!