Failure in before* leads to test skip in case of data driven testing

[vinaybalepur]

TestNG Version

Version - 6.14.2

Expected behavior

If configuration methods before* fails then test cases should not be skipped.

Actual behavior

The test cases are skipped and are not executed. This is observed even when configfailurepolicy="continue" is set in the xml file. Attaching the xml file as well. TestDepends.txt Testxml.txt

Is the issue reproductible on runner?

• [ ] Shell
• [ ###] Maven
• [ ] Gradle
• [ ] Ant
• [ ] Eclipse
• [ ] IntelliJ
• [ ] NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue Has been shared

[krmahadevan]

@vinaybalepur - TestNG is working here as designed. A failure in a configuration method will cause a @Test method to skip. There's no way of circumventing that rule because that is what configuration methods are there for. They basically help you define the entry and exit criteria for a test.

The attribute configfailurepolicy lets you tell TestNG, whether TestNG should attempt at executing a configuration method once again, after its failed already once. Its not meant to serve the purpose of what you are looking at.

Closing this issue.

[vinaybalepur]

Thank you for the clarification. But this is not the use of I have read. Please find the testng documentation of the same.

configfailurepolicy	skip\	continue	Whether TestNG should continue to execute the remaining tests in the suite or skip them if an @Before* fails. Default behavior is skip.

[krmahadevan]

@vinaybalepur - Ok my bad. I got confused with whats written in the DTD. What you called out and what's in the documentation is correct. I was wrong.

But TestNG is working fine. Here's a sample.

I have basically 3 test classes

1. SampleTestClassWithMultipleMethods - A test class with a failing config method and housing multiple @Test methods.

import org.testng.Reporter;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import java.lang.reflect.Method;

public class SampleTestClassWithMultipleMethods {
    @BeforeMethod
    public void beforeSampleTestClassWithMultipleMethods(Method method) {
        if ("a".equalsIgnoreCase(method.getName())) {
            throw new RuntimeException("throwing exception for SampleTestClassWithMultipleMethods");
        }
    }

    @Test
    public void a() {
        Reporter.log(getClass().getSimpleName() + ".a()");
    }

    @Test
    public void b() {
        Reporter.log(getClass().getSimpleName() + ".b()");
    }
}

2. SampleTestClassWithMultipleInvocations - A test class with a failing config method and houses a @Test method that has multiple invocation counts.

import org.testng.Reporter;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import java.util.concurrent.atomic.AtomicInteger;

public class SampleTestClassWithMultipleInvocations {
    private AtomicInteger counter = new AtomicInteger(0);

    @BeforeMethod
    public void beforeSampleTestClassWithMultipleInvocations() {
        if (counter.incrementAndGet() == 1) {
            throw new RuntimeException("throwing exception for SampleTestClassWithMultipleInvocations");
        }
    }

    @Test(invocationCount = 3)
    public void a() {
        Reporter.log(getClass().getSimpleName() + ".a() ==>" + counter.get());
    }
}

3. SampleTestClassWithDataDrivenTestMethod - A test class with a failing config method and houses a @Test method which is powered by a data provider.

import org.testng.Reporter;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

import java.lang.reflect.Method;

public class SampleTestClassWithDataDrivenTestMethod {

    @BeforeMethod
    public void beforeSampleTestClassWithDataDrivenTestMethod(Method method, Object[] params) {
        String value = params[0].toString();
        if ("one".equalsIgnoreCase(value)) {
            throw new RuntimeException("throwing exception for SampleTestClassWithDataDrivenTestMethod");
        }
    }

    @Test(dataProvider = "dp")
    public void a(String a) {
        Reporter.log(getClass().getSimpleName() + ".a() ==> " + a);
    }

    @DataProvider(name = "dp")
    public Object[][] getData() {
        return new Object[][]{
                {"one"},
                {"two"},
                {"three"}
        };
    }
}

I have a listener that looks like below

import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;
import org.testng.Reporter;

import java.util.LinkedList;
import java.util.List;

public class LocalListener implements IInvokedMethodListener {
    private List<String> logs = new LinkedList<>();
    @Override
    public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {

    }

    @Override
    public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
        logs.addAll(Reporter.getOutput(testResult));
    }

    public List<String> getLogs() {
        return logs;
    }
}

Finally I have a test runner class, that looks like below

import org.assertj.core.api.Assertions;
import org.testng.ITestNGListener;
import org.testng.TestNG;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlSuite;
import org.testng.xml.XmlTest;

import java.io.File;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

public class TestRunner {
    @BeforeClass
    public void beforeClass() {
        File file = new File(TestNG.class.getProtectionDomain().getCodeSource().getLocation().getFile());
        System.err.println("Working with TestNG version :" + file.getParentFile().getName());
    }
    @Test(dataProvider = "dp")
    public void testMethod(XmlSuite.FailurePolicy policy, List<String> expected) {
        TestNG testng = new TestNG();
        XmlSuite suite = newXmlSuite();
        testng.setXmlSuites(Collections.singletonList(suite));
        LocalListener listener = new LocalListener();
        testng.addListener((ITestNGListener) listener);
        testng.setConfigFailurePolicy(policy);
        System.err.println("Working with the XML");
        System.err.println(suite.toXml());
        testng.run();
        Assertions.assertThat(listener.getLogs()).containsExactlyElementsOf(expected);
        if (listener.getLogs().isEmpty()) {
            System.err.println("No Logs available for failure policy [" + policy + "]");
        } else {
            System.err.println("Method logs for failure policy [" + policy + "]");
            listener.getLogs().forEach(System.err::println);
        }
    }

    @DataProvider(name = "dp")
    public Object[][] getData() {
        return new Object[][]{
                {XmlSuite.FailurePolicy.SKIP, new ArrayList<>()},
                {XmlSuite.FailurePolicy.CONTINUE, Arrays.asList(
                        "SampleTestClassWithMultipleMethods.b()",
                        "SampleTestClassWithMultipleInvocations.a() ==>2",
                        "SampleTestClassWithMultipleInvocations.a() ==>3",
                        "SampleTestClassWithDataDrivenTestMethod.a() ==> two",
                        "SampleTestClassWithDataDrivenTestMethod.a() ==> three"
                )}
        };
    }

    private static XmlSuite newXmlSuite() {
        Class<?>[] classes = new Class[]{
                SampleTestClassWithMultipleMethods.class,
                SampleTestClassWithMultipleInvocations.class,
                SampleTestClassWithDataDrivenTestMethod.class
        };
        XmlSuite suite = new XmlSuite();
        suite.setName("My_Suite");
        XmlTest xmlTest = new XmlTest(suite);
        xmlTest.setName("My_Test");
        Arrays.stream(classes).forEach(aClass -> {
            XmlClass xmlClass = new XmlClass(aClass);
            xmlTest.getClasses().add(xmlClass);
        });
        return suite;

    }
}

And here's the output

Working with TestNG version :6.14.3
Working with the XML
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="My_Suite">
  <test thread-count="5" name="My_Test">
    <classes>
      <class name="com.rationaleemotions.github.issue1754.SampleTestClassWithMultipleMethods"/>
      <class name="com.rationaleemotions.github.issue1754.SampleTestClassWithMultipleInvocations"/>
      <class name="com.rationaleemotions.github.issue1754.SampleTestClassWithDataDrivenTestMethod"/>
    </classes>
  </test> <!-- My_Test -->
</suite> <!-- My_Suite -->

===============================================
My_Suite
Total tests run: 8, Failures: 0, Skips: 8
Configuration Failures: 3, Skips: 5
===============================================

No Logs available for failure policy [skip]
Working with the XML
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="My_Suite">
  <test thread-count="5" name="My_Test">
    <classes>

      <class name="com.rationaleemotions.github.issue1754.SampleTestClassWithMultipleMethods"/>
      <class name="com.rationaleemotions.github.issue1754.SampleTestClassWithMultipleInvocations"/>
      <class name="com.rationaleemotions.github.issue1754.SampleTestClassWithDataDrivenTestMethod"/>
    </classes>
  </test> <!-- My_Test -->
</suite> <!-- My_Suite -->

===============================================
My_Suite
Total tests run: 8, Failures: 0, Skips: 3
Configuration Failures: 3, Skips: 0
===============================================

Method logs for failure policy [continue]
SampleTestClassWithMultipleMethods.b()
SampleTestClassWithMultipleInvocations.a() ==>2
SampleTestClassWithMultipleInvocations.a() ==>3
SampleTestClassWithDataDrivenTestMethod.a() ==> two
SampleTestClassWithDataDrivenTestMethod.a() ==> three

===============================================
Default Suite
Total tests run: 2, Failures: 0, Skips: 0
===============================================

In your example the configuration method that is failing is a @BeforeClass method, which gets executed ONLY once per test class. It doesn't get retried at all when TestNG is trying to execute the next @Test method because it gets executed only once for all the @Test methods that reside in a test class. That is why you see, TestNG not retrying anything even though you try to use the configfailurepolicy as continue

Hope that adds clarity.