Retry mechanism

[glureau-betclic]

I'd love to have a mechanism that retries failed tests but not with the rerun file mechanism. I'm interested to have a report that is OK if at least one of the X attempts is OK (retry a test only when it fails indeed), and generate a unique report files. (Better again if the report can flag the flaky tests.)

Is something already available for this use case?

If no, I think about forking this repository to be able to implement my own AndroidFeatureRunner + RunNotifier, to be able to call child.run(myNotifier) as much as I need. Is it a good way to approach the problem?

Thanks for your time and this great tool!

[lsuski]

I think that there is no such feature currently. Maybe in AndroidJunitRunner from which CucumberAndroidJUnitRunner inherits but I doubt. In my opinion this feature could be implemented independently of cucumber and based on junit. From perpective of junit and other tools relying on it Cucumber feature is treated as class and scenario as method so I would research if something similar is available in Android or junit world. If you plan to implement it then you can make PR and we could try to integrate it in this repo

[glureau-betclic]

So far I've seen that Maven Surefire defines a rerun mechanism, Gradle does (or plan to do) the same, but both tools are not well suited when we use Cucumber on a device farm. In my case for example, I'd have to upload the apks (instrumentation+app under tests) as much as I want retries. And when using Bitrise, which generates only one Firebase test token by workflow, I'd have to define multiple workflows or recursively chaining the same workflow, clearly something that I want to avoid.

https://github.com/apache/maven-surefire/pull/249 https://blog.gradle.org/gradle-flaky-test-retry-plugin

There is a JUnit 5 rerunner project here https://github.com/artsok/rerunner-jupiter/ but so far Cucumber is locked on JUnit4 so I'm discarding this option for now as I'm not sure what you're planning on this upgrade.

On JUnit4, there is a TestRule class that can be define next to your annotated [@]Test methods to define the retry behaviour: https://stackoverflow.com/a/8301639/1898596 or https://wiki.saucelabs.com/display/DOCS/How+to+Deal+with+Flaky+Java+Tests It has some drawbacks as far as I understand:

• not sure it's possible to replay the [@]Setup method, so I suppose the Background in Cucumber, and that's indeed quite problematic if we want a proper restart.
• it pushes the question of the expected behaviour when used in combination with Orchestrator, as the instrumentation app is restarted, rerun could potentially run on another device (sharding).

Having the app fully restarted improves the test stability so it could be great to handle the retry mechanism in such a way it's Orchestrator/Spoon compatible.

Eventually another cool thing could be to detect the retry count directly from a tag on the test itself, so that we can mark some tests as flaky with some details about how to retry them. Can Gherkin offer a proper way to pass data on a tag for this purpose? (Personally I'm great with a global settings for now)

As my understanding of JUnit intricacies are very limited, feel free to push any information or ideas here 🙏

[glureau-betclic]

I hacked something this evening, looks promising so far (I mean it works in my simple case :) ) https://github.com/glureau-betclic/cucumber-android/commit/5653931b953e47c68312c047b8a2f0a7514a95fb?w=1

If you've some time to review it, I'd love some feedback on the approach before to go further.

And if the approach with custom AndroidFeatureRunner + RunListener is viable, I'd love a list of stuff I'd have to support to make this feature mergeable (scenario, outline, ...).