search

gradle / test-retry-gradle-plugin

Gradle plugin to retry tests that have failed to mitigate test flakiness.
Apache License 2.0
222 stars 50 forks source link

TestNg Cucumber Test Suite is repeated as a whole #164

Open realdadfish opened 1 year ago

realdadfish commented 1 year ago

We're using cucumber-testng-7.8.1 together with testng-7.6.1 and try to setup test retry with this Gradle plugin.

This is one test class for a big set of tests that are largely divided by Cucumber tags:

@CucumberOptions(
    features = [
        "src/test/resources/features/"
    ],
    glue = [
        "our.glue.code",
        "more.shared.glue.code"
    ],
    plugin = [
        "pretty",
        "io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm",
        "junit:build/reports/junit/test-results.xml",
        "json:build/reports/cucumber/cucumber-json-report.json"
    ],
    tags = "@dab-gsd and (not @ignore)"
)
@Test
@Listeners(TestLifecycleListenerGsd::class)
class TestRunnerGsd : AbstractTestNGCucumberTests()

Now we figured that when a single test fails not the single test is re-executed, but the whole TestRunnerGsd at once. We've configured no class filters or the like, just enabled the plugin and set maxRetries to 3.

Is retrying Cucumber tests on a per-scenario level supported by this plugin?

leonard84 commented 9 months ago

It does not support TestNG and might not fit your use-case, but if you just want individually retry-able cucumber tests, you might want to take a look at https://github.com/gradle/cucumber-companion.

pshevche commented 4 months ago

I looked briefly into it. You are right; at the moment, all scenarios in the feature will be retried. This is caused by how TestNG executes and reports individual scenarios: https://github.com/prasanta-biswas/testng-cucumber/blob/master/src/main/java/cucumber/api/testng/AbstractTestNGCucumberTests.java#L25-L28. Basically, it uses scenarios as parameters for a data-driven test. So, TestNG runs a single method executing all scenarios. Currently, the plugin will retry the declared method including all of its iterations (i.e., all scenarios in case of TestNG).

I don't have a good recommendation at the moment apart from splitting large feature files into multiple. We have an issue to revisit Cucumber support on our roadmap, but there is no schedule for it yet: https://github.com/gradle/test-retry-gradle-plugin/issues/279. We will check if we can provide a better support for the TestNG runner in that investigation.