(InvocationCount or DataProvider) and RetryAnaylzer when used togther performs incorrect behaviour

[shroffsagar]

Below are the steps to reproduce both the issue. For data provider issue, one needs to use a data provider with testinputs same as invocationCount.

@UseAsTestcaseName("Sanity-TC01010-test")
@MaxRetryCount(3)
@Test(groups="sanity test", invocationCount=2) // or a data provider with 2 inputs
public void test(){
    System.out.println("sample test : "+Thread.currentThread().hashCode());
    Assert.fail();
}

Above configures my RetryAnalyzer to max retry attempts to 3. And total invocationCount to 2 Below is the output

sample test : 2041228125
sample test : 2041228125
sample test : 2041228125
Completed retry
sample test : 2041228125
sample test : 2041228125
sample test : 2041228125
Completed retry
sample test : 2041228125
sample test : 2041228125
sample test : 2041228125
Completed retry

As noticed in the above output. Test is invoked more then twice.

The total invocation goes wrong only when a retry is invoked.

The current invocation depends on the total times the method is invoked. The count is cumulative with each new iteration.

Please review below examples. All the retry fails in below examples

@MaxRetryCount(4)
invocationCount=1 // or a data provider with 1 inputs
Total invocations made = 1

@MaxRetryCount(4)
invocationCount=2 // or a data provider with 2 inputs
Total invocations made = 3

@MaxRetryCount(4)
invocationCount=3 // or a data provider with 3 inputs
Total invocations made = 6

@MaxRetryCount(4)
invocationCount=4 // or a data provider with 4 inputs
Total invocations made = 10

[krmahadevan]

I have tried reproducing this in TestNG 7.8.0 and I am not able to.

Using invocation counts with retry analyzer

import java.util.concurrent.atomic.AtomicInteger;
import org.testng.Assert;
import org.testng.IExecutionListener;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(CountingListener.class)
public class InvocationCountPoweredSampleTestCase {

  private static final AtomicInteger counter = new AtomicInteger(0);

  @Test(retryAnalyzer = OnceMore.class, invocationCount = 2)
  public void testMethod() {
    counter.incrementAndGet();
    int currentInvocationCount = Reporter.getCurrentTestResult().getMethod()
        .getCurrentInvocationCount();
    System.err.println(getClass().getName() + ".testMethod(): Counter = " + counter.get() +", Invocation Count = " + currentInvocationCount);
    Assert.fail();
  }

  public static class OnceMore implements IRetryAnalyzer {

    private final AtomicInteger count = new AtomicInteger(0);

    @Override
    public boolean retry(ITestResult result) {
      int currentInvocationCount = result.getMethod().getCurrentInvocationCount();
      int current = count.incrementAndGet();
      System.err.println(getClass().getEnclosingClass().getName() + ".retry(): Counter = " + counter.get() +", Invocation Count = " + currentInvocationCount);
      return current < 2;
    }
  }

  public static class CountingListener implements IExecutionListener {

    @Override
    public void onExecutionFinish() {
      System.err.println(getClass().getEnclosingClass().getName() + ": Total count for testMethod() is " + counter.get());
    }
  }
}

Output

com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(): Counter = 1, Invocation Count = 0
com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.retry(): Counter = 1, Invocation Count = 0

Test ignored.
com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(): Counter = 2, Invocation Count = 1
com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.retry(): Counter = 2, Invocation Count = 1

java.lang.AssertionError: null

    at com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(InvocationCountPoweredSampleTestCase.java:24)

com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(): Counter = 3, Invocation Count = 2
com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.retry(): Counter = 3, Invocation Count = 2

java.lang.AssertionError: null

    at com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(InvocationCountPoweredSampleTestCase.java:24)

com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(): Counter = 4, Invocation Count = 3
com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.retry(): Counter = 4, Invocation Count = 3

java.lang.AssertionError: null

    at com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(InvocationCountPoweredSampleTestCase.java:24)

===============================================
Default Suite
Total tests run: 4, Passes: 0, Failures: 3, Skips: 0, Retries: 1
===============================================

com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase: Total count for testMethod() is 4

Process finished with exit code 0

Notice this line in the above output:

com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase: Total count for testMethod() is 4

For every test invocation's failure, the retry happens twice. So total invocations are 2*2

Using data provider with retry analyzer

import java.util.concurrent.atomic.AtomicInteger;
import org.testng.Assert;
import org.testng.IExecutionListener;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(CountingListener.class)
public class DataProviderPoweredSampleTestCase {

  private static final AtomicInteger counter = new AtomicInteger(0);

  @Test(retryAnalyzer = OnceMore.class, dataProvider = "dp")
  public void testMethod(int ignored) {
    counter.incrementAndGet();
    int currentInvocationCount = Reporter.getCurrentTestResult().getMethod()
        .getCurrentInvocationCount();
    System.err.println(
        getClass().getName() + ".testMethod(): Counter = " + counter.get() + ", Invocation Count = "
            + currentInvocationCount);
    Assert.fail();
  }

  @DataProvider(name = "dp")
  public Object[][] getTestData() {
    return new Object[][]{
        {1},
        {2}
    };
  }

  public static class OnceMore implements IRetryAnalyzer {

    private final AtomicInteger count = new AtomicInteger(0);

    @Override
    public boolean retry(ITestResult result) {
      int currentInvocationCount = result.getMethod().getCurrentInvocationCount();
      int current = count.incrementAndGet();
      System.err.println(
          getClass().getEnclosingClass().getName() + ".retry(): Counter = " + counter.get()
              + ", Invocation Count = " + currentInvocationCount);
      return current < 2;
    }
  }

  public static class CountingListener implements IExecutionListener {

    @Override
    public void onExecutionFinish() {
      System.err.println(
          getClass().getEnclosingClass().getName() + ": Total count for testMethod() is "
              + counter.get());
    }
  }
}

Output

com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.testMethod(): Counter = 1, Invocation Count = 0
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.retry(): Counter = 1, Invocation Count = 0

Test ignored.
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.testMethod(): Counter = 2, Invocation Count = 1
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.retry(): Counter = 2, Invocation Count = 1

java.lang.AssertionError: null

    at com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.testMethod(DataProviderPoweredSampleTestCase.java:27)

com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.testMethod(): Counter = 3, Invocation Count = 2
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.retry(): Counter = 3, Invocation Count = 2

Test ignored.
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.testMethod(): Counter = 4, Invocation Count = 3
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.retry(): Counter = 4, Invocation Count = 3

java.lang.AssertionError: null

    at com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.testMethod(DataProviderPoweredSampleTestCase.java:27)

===============================================
Default Suite
Total tests run: 4, Passes: 0, Failures: 2, Skips: 0, Retries: 2
===============================================

com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase: Total count for testMethod() is 4

Process finished with exit code 0

Please note the below line in the above output

com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase: Total count for testMethod() is 4

If this is still a problem, please help share a reproducible sample and comment.