Try out junit5 @RepeatTest tag to replace the RetryTestRule

[sv-ohorvath]

I came across junit 5 "@RepeatedTest" that would allow repeating a specific test as many times as we want, going through the whole lifecycle of a test, meaning it will call the tearDown methods and clear the app in between runs. It could be a better alternative to the RetryTestRule and the for loops we use ATM (but not entirely sure) if we can set it to repeat only if the first one fails: https://www.baeldung.com/junit-5-repeated-test

[sv-ohorvath]

As I read more, the RepeatedTest tag will just repeat tests n-times without a condition (for example we want them to repeat just in case a fail happens). It can be added as a meta-annotation for other custom annotations, but we cannot add some kind of implementation to these custom annotations (they don't allow it). It also requires adding JUnit 5 dependencies.

I've also tried this solution from S.O, in addition to our RetryTestRule: https://stackoverflow.com/questions/8295100/how-to-re-run-failed-junit-tests-immediately/64731321#64731321 - but while running it, it only restarted the application without resetting the app state as we need.

Another solution that I've found was a Gradle retry plugin: https://github.com/gradle/test-retry-gradle-plugin But it turns out it is not applicable for android instrumented tests, just unit tests: https://github.com/gradle/test-retry-gradle-plugin/issues/17

For now, the only other solution I think would be at hand to experiment with, is to increase the num-of-flaky-test-attempts in the flank config file to have more retries on Firebase

• slightly decreasing the chances for a flaky test to fail 3 times and block a PR/build (as opposed to 2 times as we have now).
• basically, it is just like adding a rule to retry in case a fail happens
• this would clear the app state between retries, as we want

@AaronMT I've documented here what we've already been discussing. My investigation is kind of stuck, for now. Let me know what you think, in case you have other suggestions, and if my assumption about the last point is wrong or maybe I'm missing something.

[AaronMT]

Thank you for investigating this. Let's revisit if we find anything else or see what other projects are using.