successPercentage attribute of @Test does not work correctly

[gdong42]

I'm using testng-6.8, while running a multiple invocation test, I found that the "successPercentage" of @Test annotation is not working correctly.

Reproducer:

===build.xml===

...
<target name="testRepeat">
    <testng classpathref="test.cp" outputDir="out" 
      haltOnFailure="false" verbose="2" >
      <xmlfileset dir="." includes="repeat.config.xml"/>
    </testng>    
</target>
...

===repeat.config.xml===

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="repeat-suite" verbose="1" >
  <test name="repeat-test" >
    <classes>
       <class name="testng.samples.RepeatTest"  />
    </classes>
  </test>
</suite>

===RepeatTest.java===

package testng.samples;

import org.testng.annotations.Test;
import org.testng.Assert;

public class RepeatTest {

  int count = 0;

  @Test
  public void normal() {
    System.out.println("repeat..." + (++count));
  }

  @Test(invocationCount = 30, successPercentage=98)
  public void successRate() {
    System.out.println("repeat..." + (++count));
    if (count > 15) {
      Assert.fail("fail with count > 15");
    }   
  }
}

According to http://testng.org/doc/documentation-main.html#annotations, if "successPercentage" is specified, testng should fail the test if pass percentage is smaller than expected. However, if you run above reproducer, you will see it prints below output, but never sends a FAILURE event to listeners. It only works when percentage is set to 100, or is not set.

This seems like a regression, I did no see the issue in testng-5.8.

===output===

   [testng] PASSED: successRate
   [testng] PASSED: successRate
   [testng] 
   [testng] ===============================================
   [testng]     repeat-test
   [testng]     Tests run: 31, Failures: 0, Skips: 0
   [testng] ===============================================
   [testng] 
   [testng] 
   [testng] ===============================================
   [testng] repeat-suite
   [testng] Total tests run: 31, Failures: 16, Skips: 0
   [testng] ===============================================
   [testng] 

Notice that failure count in repeat-test section is 0, and it did not log any FAILURE even it does not meet the expected percentage.

Regards, Gan

[robertrv]

This is a very old issue but still standing, just tested locally. Any idea if is going to be addressed or any workaround?

[pcl]

@cbeust : this is biting us when we use TestNG together with Jenkinsfile-based builds. Any thoughts about how hard it'd be to fix, or appetite for accepting a PR?

[cbeust]

Always happy to accept a PR, Patrick!

[joshua-strickland]

I am also seeing behavior where tests that succeed below the specified success % are not causing a failure in the suite.

Currently using TestNG v6.10

This seems like a fairly old ticket, maybe there has been a fix for this since it's been open? Are there any plans to fix this? Thanks

[krmahadevan]

Revisited this issue using TestNG 7.1.0. Works fine and cant reproduce the problem.

Java class

package com.rationaleemotions;

import org.testng.Assert;
import org.testng.annotations.Test;

public class RepeatTest {

  int count = 0;

  @Test
  public void normal() {
    System.out.println("repeat..." + (++count));
  }

  @Test(invocationCount = 30, successPercentage = 98)
  public void successRate() {
    System.out.println("repeat..." + (++count));
    if (count > 15) {
      Assert.fail("fail with count > 15");
    }
  }
}

suite file

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="309_suite" parallel="false" configfailurepolicy="continue">
  <test name="309_test">
    <classes>
      <class name="com.rationaleemotions.RepeatTest"/>
    </classes>
  </test>
</suite>

Output

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] com.rationaleemotions.RepeatTest.successRate
[INFO]   Run 1: PASS
[INFO]   Run 2: PASS
[INFO]   Run 3: PASS
[INFO]   Run 4: PASS
[INFO]   Run 5: PASS
[INFO]   Run 6: PASS
[INFO]   Run 7: PASS
[INFO]   Run 8: PASS
[INFO]   Run 9: PASS
[INFO]   Run 10: PASS
[INFO]   Run 11: PASS
[INFO]   Run 12: PASS
[INFO]   Run 13: PASS
[INFO]   Run 14: PASS
[ERROR]   Run 15: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 16: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 17: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 18: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 19: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 20: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 21: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 22: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 23: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 24: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 25: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 26: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 27: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 28: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 29: RepeatTest.successRate:19 fail with count > 15
[ERROR]   Run 30: RepeatTest.successRate:19 fail with count > 15
[INFO]
[INFO]
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.197 s
[INFO] Finished at: 2020-03-14T12:14:16+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test (default-test) on project github_309: There are test failures.
[ERROR]
[ERROR] Please refer to /Users/krmahadevan/temp/github_309/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Test project that was used.

github_309.zip