onTestSkipped method of listener is called but it is marked as failed in the report

[gruchika]

Encountered a problem when exception is thrown in the dataprovider, onTestSkipped method of listener is called but it is marked as failed in the report(ITestResult object FAILURE status)

package testngPackage;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
import static org.testng.Assert.assertTrue;

/**
- If the exception is encountered in the dataprovider
  *
  */
  public class SkippedAndFailedTest {

  @DataProvider(name = "Skipped&FailedTest")
  public Object[][] createData3() {  
      Object[][] obj = new Object[][] {
                 { "Welcome", new Integer(01) },
                 };
      try {
          if(obj != null)
              throw new Exception("Exception in the Dataprovider");
      } catch (Exception e) {
          assertTrue(false, e.getMessage());
      }
      return obj;
  }
  @Test(dataProvider = "Skipped&FailedTest")
  public void verifyData3(String n1, Integer n2) {
   System.out.println(n1 + " " + n2);
  } 
  }

onStart

invoked onTestSkipped

onFinish

Suite

Total tests run: 1, Failures: 1, Skips: 0

IN generateReport

--- resultMap based on the test status ===> [TestResult name=verifyData3 status=FAILURE method=SkippedAndFailedTest.verifyData3(java.lang.String, java.lang.Integer)[pri:0, instance:testngPackage.SkippedAndFailedTest@721708f9] output={null}]

[trijimonpr]

The issue is reproducible for me as well. Skipped tests are being displayed as failed by SuiteRunnerWorker.java. Below is a sample output that confirms the behaviour:

TEST_API
Tests run: 2, Failures: 0, Skips: 2

TEST_API_Suite

Total tests run: 2, Failures: 2, Skips: 0

I did some analysis on the flow, here is what I have found so far:

In the Invoker.invokeTestMethods(..) method, the current test's result status is set to skipped(3) from failed(2) when there are errors in DP by registerSkippedTestResult() method but the test context(m_testContext) is not being updated anywhere. Since the final suite level stats are being pulled from test context, the skipped tests are being shown as failed ones.

[Tibor17]

We have found this issue when one of our user reported a bug against apache/maven-surefire-plugin. Obviously this bug belongs to TestNG. https://github.com/apache/maven-surefire/pull/70 We found that the TestNG 5.x marked the test with a failure if DataProvider thew an exception. Currently TestNG 6.8.8 changed the behavior and listener calls onTestSkipped which results in situations where people report against apache/maven-surefire-plugin. Are you willing to keep the behavior of TestNG 5.x in regard of this usecase please?