Retry a test until first fail

MarathonLabs / marathon

Cross-platform test runner

https://docs.marathonlabs.io

GNU General Public License v2.0

584 stars 122 forks source link

Retry a test until first fail #329

Open aartikov opened 4 years ago

aartikov commented 4 years ago

Is it possible to retry a test until first fail in a strict mode? It will be useful to investigate flaky tests. At least until this issue #322 is fixed.

Malinskiy commented 4 years ago

What if the test would not fail for 1000 executions?

Any reason why we can't use strict mode with a specific number of executions like 100? This way you have at list a limitation of the number of executions and have the control over the precision of calculated probability:

if you have 100 executions then it's at least 1% precision
if you have 1000 executions then it's at least 0.1% precision and so on

aartikov commented 4 years ago

@Malinskiy I want to use marathon to catch a rare crash on CI emulator. I need logcat logs for this crash. I can run a test 100 times and a crash will arise with a great chance. But the logs will be overwritten because of this issue #322. This is my main problem. The other problem - 100 is not optimal value. Maybe 50 times is enough to not waste resources of CI server. I don't know beforehand.

ZakTaccardi commented 4 years ago

this seems to be related to #360

a variation on strictMode that bails out immediately when the first failure is encountered would be very useful

ZakTaccardi commented 2 years ago

note: use case here is that I want to rerun a test up to X times to investigate if there is any flakiness. If encountered, I want to be able to review the marathon report for that failure to debug the issue.

When the last test succeeds, the marathon report looks successful - so I'm unable to debug the failed test to identify the flakiness.

gmk57 commented 1 year ago

Huge plus to this proposal.

TBH, I don't quite understand the use case for strictMode in its current shape (without fail-fast behavior):

For regular CI builds we typically want to "ignore" flakiness, so we use countSharding + strictMode: false
To investigate flakiness (or verify it's fixed) we typically run one particular test "manually" many times (locally or on CI). Repeating after first failure is just a waste of resources in this case. If we specifically wanted to measure the percentage of failures, we could turn off strictMode as well (build success or failure doesn't matter much here, stats are visible anyway).

Malinskiy commented 1 year ago

Fail fast is now support via https://docs.marathonlabs.io/intro/configure#execution-strategy, as well as success fast depending on the execution mode selected.

Keep in mind that there are additional requirements now on runs mentioned in the docs.

The fast flag operates on a test-by-test basis, so it fails/succeeds the test, but not the whole run