gradle / test-retry-gradle-plugin

Gradle plugin to retry tests that have failed to mitigate test flakiness.
Apache License 2.0
225 stars 50 forks source link

Is there a way to get the retry number 1,2,3... during run time inside the gradle test task? #264

Closed BugBuster123 closed 6 months ago

BugBuster123 commented 7 months ago

Currently I am using a cucumber framework for automation. I have been running the failed cases using the retry gradle plugin. Please find the sample gradle task below

task failedtestrunner(type: Test) { systemProperty("cucumber.plugin", "html:reports/$browser/${name}/cucumber.html") include "**/FailedTestRunner.class" testLogging.showStandardStreams = true retry { maxRetries = 5 failOnPassedAfterRetry = false } }

The problem that I am facing is when the task is rerun the reports folder gets overwritten every time. I will get only the result of the last run in the reports folder. If there is a way to get the retry number during run time I can append it to the reports folder (just like browser and task name given in the example above) and can get access to all the failed reports.

pshevche commented 6 months ago

@BugBuster123 , have you explored Gradle Build Scans for your use case: https://scans.gradle.com/? Those provide a more user-friendly reporting than the default XML/HTML report. In particular, they provide detailed information about each test run, as well as the overall outcome. Here is an example of a flaky test in JUnit: https://ge.junit.org/s/tcwmbmjybjktg/tests/task/:junit-jupiter-engine:test/details/org.junit.jupiter.engine.extension.TimeoutExtensionTests$SeparateThread/oneThreadStuckForeverAndOtherTestsInSameThread()?top-execution=1

We don't have immediate plans to enhance the reporting of retries, considering that Build Scans provide a much more superior visualization. Feel free to re-open the issue if you feel like your use case is not covered by Build Scans.