If you throw a skip exception in one of your listeners whilst a BeforeX method has already failed, the state of the tests should be skipped (in 7.4.0 it was the case. The new skip overrode the configure issue)
Also we should not lose parameter/data provider data.
Actual behavior
Since 7.5 (Worked fine in 7.4.0 (despite #2522 )) If you throw a skip exception in one of your listeners whilst a BeforeX method has already failed, the state of the tests will switch to "Failed".
Another consequence of this is that if you use a data provider, the parameter will be lost in the ITestResult object .
Is the issue reproducible on runner?
[ ] Shell
[x] Maven
[ ] Gradle
[ ] Ant
[x] Eclipse
[x] IntelliJ
[ ] NetBeans
Test case sample
In my examples bellow, My listener, DP_Listeners, throws a skip exception.
My test class MyTest has a beforeSuite method that contains a failing assertion.
(I have omitted my helper methods, but can add them if you need it)
@Test
public void problemTest() {
// Rampup
TestNG myTestNG = createTestNG();
TestListenerAdapter tla = fetchTestResultsHandler(myTestNG);
// Define suites
XmlSuite mySuite = addSuitToTestNGTest(myTestNG, "Problem Test Suite");
// Add listeners
mySuite.addListener(DP_Listeners.class.getTypeName());
// Create an instance of XmlTest and assign a name for it.
XmlTest myTest = attachTestToSuite(mySuite, "Test config methods");
myTest.setXmlClasses(Arrays.asList(new XmlClass(MyTest.class)));
myTestNG.run();
SoftAssert softAssertion = new SoftAssert();
softAssertion.assertEquals(tla.getPassedTests().size(), 0, "No tests should be passed");
softAssertion.assertEquals(tla.getSkippedTests().size(), 2, "All tests tests should be skipped");
softAssertion.assertEquals(tla.getFailedTests().size(), 0, "No tests tests should be failed");
if (tla.getFailedTests().size()>0) {
softAssertion.assertEquals(tla.getFailedTests().get(0).getParameters().length, 1, "We should still keep our parameters");
}
softAssertion.assertAll();
}
My Listener
public class DP_Listeners implements ITestListener {
@Override
public void onTestStart(ITestResult result) {
if (result.getName().equals("simpleTest1")) {
throw new SkipException("Forced Skip");
}
}
}
My target test class:
public class MyTest {
@BeforeSuite
public void beforePhasedSuite() {
int beforeValue = 13;
Assert.assertEquals(beforeValue, 14);
}
@DataProvider (name = "myProvider")
public Object[][] dpMethod() {
return new Object [][] { {"P1"}};
}
@Test(dataProvider = "myProvider")
public void simpleTest1(String s) {
System.out.println("In test1 "+s);
}
@Test(dataProvider = "myProvider")
public void simpleTest2(String s) {
System.out.println("In test2 "+ s);
}
}
@baubakg - I was not able to reproduce the error using 7.4.0. Please find attached a sample project for the same. Can you please confirm ?
github_2721.zip
TestNG Version
7.5 (regression in regards to 7.4.0)
Expected behavior
If you throw a skip exception in one of your listeners whilst a BeforeX method has already failed, the state of the tests should be skipped (in 7.4.0 it was the case. The new skip overrode the configure issue)
Also we should not lose parameter/data provider data.
Actual behavior
Since 7.5 (Worked fine in 7.4.0 (despite #2522 )) If you throw a skip exception in one of your listeners whilst a BeforeX method has already failed, the state of the tests will switch to "Failed".
Another consequence of this is that if you use a data provider, the parameter will be lost in the ITestResult object .
Is the issue reproducible on runner?
Test case sample
In my examples bellow, My listener, DP_Listeners, throws a skip exception.
My test class MyTest has a beforeSuite method that contains a failing assertion.
(I have omitted my helper methods, but can add them if you need it)
My Listener
My target test class: