gradle / test-retry-gradle-plugin

Gradle plugin to retry tests that have failed to mitigate test flakiness.
Apache License 2.0
222 stars 50 forks source link

Implement CI fanout for Gradle cross version tests #301

Closed pshevche closed 1 month ago

pshevche commented 2 months ago

Description

For the release of the plugin, we require a successful "Verify All" job for the release commit. This job depends on 3 other jobs:

While the first two jobs are relatively quick (both taking around 8m), the last one is painfully slow, taking between 4 and 7 hours to complete based on the agent used. This effectively means that we cannot complete a release within a day, as this build takes the majority of the work day.

I suggest splitting this job into multiple ones, which will run tests only for a certain major Gradle version. This would allow us to run them in parallel and hopefully cut the waiting time by approximately 75% from 6 hours to 1.5.

Using Test Distribution is not an option in the current setup. We have very few test classes with not that many test methods. However, Gradle versions are fed as test parameters, meaning that for the "Releases" job, each method has 35 iterations. And those will be scheduled as a single unit by TD.