Feature Request: Retry on Success

[pdb-stripe]

Can you add a feature to allow retry on success? This would be useful for identifying the reliability of a test (what % does it pass/fail, not simply flaky/failing/passing).

This would likely entail adding a property to the extension, such as retryOnSuccess (and potentially maxSuccesses), as well as passing this value to the RetryTestResultProcessor for use on test runs.

Thanks!

[marcphilipp]

Thank you for submitting this feature request, @pdb-stripe! We have no immediate plans of adding such a feature and would first have to discuss it internally. For now, I'd recommend using a test framework mechanism such as @RepeatedTest for JUnit Jupiter or @RepeatUntilFailure from Spock for re-running a passing tests that you suspect to be flaky.

[pdb-stripe]

@marcphilipp thanks for the update! Do you have any recommendation for use with JUnit4/Android?

[marcphilipp]

I don't think there's a published library for this, but you should be able to achieve the desired behavior using a custom TestRule, as described in this post.

[pshevche]

We discussed this internally and decided that we'd like to keep the plugin focused on retrying only failures. For "retry until failure" functionality, please use features provided by the testing framework used.