search

EnricoMi / publish-unit-test-result-action

GitHub Action to publish unit test results on GitHub
Apache License 2.0
601 stars 178 forks source link

Help supporting flaky tests using Gradle #604

Open knsonier opened 3 months ago

knsonier commented 3 months ago

I have a java project that is using Gradle and "org.gradle.test-retry" for retrying failed tests. From the Gradle documentation it looks like there is an option called mergeReruns (https://docs.gradle.org/7.3/userguide/java_testing.html#mergereruns) that:

When mergeReruns is enabled, if a test fails but is then retried and succeeds, its failures will be recorded as instead of , within one . This is effectively the reporting produced by the surefire plugin of Apache Maven™ when enabling reruns. If your CI server understands this format, it will indicate that the test was flaky. If it does not, it will indicate that the test succeeded as it will ignore the information. If the test does not succeed (i.e. it fails for every retry), it will be indicated as having failed whether your tool understands this format or not.

However, I am still having problems as when the check run is created, it still produces an X and the test fails on a flaky. I referenced this issue here and it seems like this should be supported? One thing I noticed that seemed different was the Job Summary usually had a link in it which would take me to the failure but now it does not.

Before enabling mergeReruns I would get something like this: image

And after I enabled mergeReruns I see the "For more details" line disappeared image

I can not figure out what I'm doing incorrect here. Any help would be appreciated! Thank you.

EnricoMi commented 3 weeks ago

For that to understand it would help to look into the relevant bits of the test results XML file. First I would identify the tests that fail, second I would extract the relevant <testcase> elements from all test result files.