Add re-run functionality for flaky tests

[Bersh]

In my project I have a challenge to deal with with flaky tests - when test may fail randomly because of some external conditions not because of actual error. I think the easiest way to deal with it is to implement test re-runs - in case of test failure re-run it up to some number of times and report a failure only if all runs failed.

Very basic implementation can be like this: https://github.com/Bersh/spoon-gradle-plugin/commit/83f065012f5805c115a61d5146e0a3a5bc51bc1e

Do you think it makes sense for spoon-gradle-plugin to have this functionality?

[roman-mazur]

I do not think it should be done on the gradle plugin level. Android SDK contains a FlakyTest annotation that can be used in order to specify such tests and set number or re-runs to perform. The annotation is handled by the test runner. The new Android Testing Support Library deprecates that annotation and introduces a new one. I'm not sure if support library runner performs multiple attempts to run a failed tests, but this is fairly simple to implement with JUnit rules. Here's an example: https://gist.github.com/JakeWharton/7fe7deb1f7f4a795c120

So, my opinion is not on the gradle level: doing this on JUnit runner level is much more flexible.

[Bersh]

@roman-mazur unfortunately old FlakyTest is deprecated as you mentioned and new one doesn't have the same functionality - in fact it's just used to mark tests that can be flaky it has nothing to do with actual re-runs. But JUnit rule approach might be good idea, thanks for the tip! I'll try.

[ishammoho]

HI @Bersh Just wondering whether you manage to solve this?

[SilviaChip]

I believe this is what we need to achieve the rerun on Espresso tests -> https://gist.github.com/patrickjh/a65505d53e25e5816b31bb44cbb0e492