search

gradle / test-retry-gradle-plugin

Gradle plugin to retry tests that have failed to mitigate test flakiness.
Apache License 2.0
218 stars 48 forks source link

Retrying TestNG Datadriven tests run whole test again instead of only running failed invocation #292

Closed itkhanz closed 3 weeks ago

itkhanz commented 3 weeks ago

Hi, I have noticed this behavior with gradle-retry-plugin while running DataDriven tests in TestNG that if any of the run of DataDriven Test gets failed, then instead of retrying that particular test run, it reruns the entire test with all the data again. This is a serious problem lets say if i have a data driven test in TestNG where I am supplying my test with different sets of data to invoke each time with a new data, then it will rerun the entire test again causing long time delay and unnessary retries to test invocations that although got passed in first attempt but will be retried because another test run with a different data for same test got failed.

To reproduce this issue, consider this scenario where i have a simple data driven test which gets invoked 2 times with a different data. During the second invokation of the test when the count is 2, then this test will fail. At this point, instead of retrying this particular test invocation, it retries the entire test again with all the invocations.

Steps to reproduce:

Run the below TestNG test in gradle with maxRetries set to 1.

import lombok.extern.slf4j.Slf4j;
import org.testng.Assert;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

@Slf4j
public class DataDrivenTest {

    @Test(dataProvider = "testDataProvider")
    public void test_demo(int count) {
        log.info("count: {}", count);
        if (count == 2) {
            Assert.fail("Test will fail when count is equal to 2");
        }
    }

    @DataProvider
    public Object[][] testDataProvider() {
        return new Object[][]{
                {1},
                {2}
        };
    }
}

Log output:-

Gradle suite > Gradle test > com.dcs.appium.testing.tests.DataDrivenTest > test_demo[0](1) STARTED

Gradle suite > Gradle test > com.dcs.appium.testing.tests.DataDrivenTest > test_demo[0](1) STANDARD_OUT
    2024-06-25 09:28:25.613 [INFO] DataDrivenTest.test_demo:13 - count: 1

Gradle suite > Gradle test > com.dcs.appium.testing.tests.DataDrivenTest > test_demo[0](1) PASSED

Gradle suite > Gradle test > com.dcs.appium.testing.tests.DataDrivenTest > test_demo[1](2) STARTED

Gradle suite > Gradle test > com.dcs.appium.testing.tests.DataDrivenTest > test_demo[1](2) STANDARD_OUT
    2024-06-25 09:28:25.618 [INFO] DataDrivenTest.test_demo:13 - count: 2

Gradle suite > Gradle test > com.dcs.appium.testing.tests.DataDrivenTest > test_demo[1](2) FAILED
    java.lang.AssertionError: Test will fail when count is equal to 2
        at org.testng.Assert.fail(Assert.java:111)
        at com.dcs.appium.testing.tests.DataDrivenTest.test_demo(DataDrivenTest.java:15)

Gradle suite > Gradle test > com.dcs.appium.testing.tests.DataDrivenTest > test_demo[0](1) STARTED

Gradle suite > Gradle test > com.dcs.appium.testing.tests.DataDrivenTest > test_demo[0](1) STANDARD_OUT
    2024-06-25 09:28:26.046 [INFO] DataDrivenTest.test_demo:13 - count: 1

Gradle suite > Gradle test > com.dcs.appium.testing.tests.DataDrivenTest > test_demo[0](1) PASSED

Gradle suite > Gradle test > com.dcs.appium.testing.tests.DataDrivenTest > test_demo[1](2) STARTED

Gradle suite > Gradle test > com.dcs.appium.testing.tests.DataDrivenTest > test_demo[1](2) STANDARD_OUT
    2024-06-25 09:28:26.051 [INFO] DataDrivenTest.test_demo:13 - count: 2

Gradle suite > Gradle test > com.dcs.appium.testing.tests.DataDrivenTest > test_demo[1](2) FAILED
    java.lang.AssertionError: Test will fail when count is equal to 2
        at org.testng.Assert.fail(Assert.java:111)
        at com.dcs.appium.testing.tests.DataDrivenTest.test_demo(DataDrivenTest.java:15)

---------------------------------------------------------------
|  Results: FAILURE (4 tests, 2 passed, 2 failed, 0 skipped)  |
---------------------------------------------------------------

4 tests completed, 2 failed

FAILURE: Build failed with an exception.

As you can see from the results of above execution, that it retries the whole test again instead of retrying it just once with data when count = 2 becuase this test got failed only on this value from datadriven test.

We actually have many test cases, where we are running the data driven tests multiple times, and if retry-plugin starts retrying the entire tests then this results in unncessary retries, and more execution time for the tests.

Could you please look into this and see if it is possible to retry only that test invocation that got failed instead of retrying it all? This would be very helpful. Thank you.

pshevche commented 3 weeks ago

@itkhanz , this is currently not possible due to some limitations in Gradle itself. This issue describes the problem for JUnit 5, but the iteration selection works similarly for TestNG, and the plugin can't tell Gradle to execute just a single iteration of the data-driven test method.

I'll close the issue for now, as there is nothing we can do until Gradle addresses the underlying issue..