Parameters not included in failure XML

[edwolb]

We use XML parameters to feed runtime parameters to our tests, and then use DataProviders to further iterate through different test scenarios. We've setup our framework such that a CSV file can be passed in through the XML file and used to run the test against specific data. The xml looks like this:

<suite name="Test Suite" verbose="1" parallel="tests" thread-count="10">
  <test name="Test" >
    <classes>
       <parameter name="csvFile" value="path/to/file.csv" />
           <class name="com.example.automation.Test.java" />
    </classes>
  </test>
</suite>

This works well for us, as it means we can use the same test against different sets of data, for different databases or one file for smoke testing and one for full regression. We have a generic DataProvider that we reuse that always looks for the "csvFile" parameter and creates a set of data to pass to the test using it.

The problem is that when tests fail for specific rows of data, and we try to replay only failed tests, the csvFile parameter goes missing. This results in a total failure, and the only way we can replay certain scenarios is by doing the entire test, which kills us for heavy data driven tests. How can we ensure that parameters are passed into the failed test XML files?

(Comments from Cedric): Probably an oversight in how I generate testng-failed.xml (I don't believe I look up the tag in the original testng.xml file).

[BabyRoot]

We have a similar problem ((( https://github.com/cbeust/testng/issues/449

[krmahadevan]

Tried using TestNG 6.12. Cannot reproduce the problem.

import org.testng.Assert;
import org.testng.ITestContext;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class DataDrivenTestClass {

    @Test(dataProvider = "dp")
    public void testMethod(int i) {
        if (i == 3) {
            Assert.fail();
        } else {
            Assert.assertTrue(true);
        }
    }

    @DataProvider(name = "dp")
    public Object[][] getData(ITestContext context) {
        int iterations = Integer.parseInt(context.getCurrentXmlTest().getAllParameters().get("iterators"));
        Object[][] data = new Object[iterations][1];
        for (int i = 0; i < iterations; i++) {
            data[i][0] = i + 1;
        }
        return data;
    }
}

Testrunner looks like below

import org.assertj.core.api.Assertions;
import org.testng.TestNG;
import org.testng.xml.Parser;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlSuite;
import org.testng.xml.XmlTest;
import org.xml.sax.SAXException;

import javax.xml.parsers.ParserConfigurationException;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;

public class TestRunnerForIssue235 {
    public static void main(String[] args) throws IOException, SAXException, ParserConfigurationException {
        TestNG testng = new TestNG();
        XmlSuite suite = new XmlSuite();
        suite.setName("suite");
        XmlTest test = new XmlTest(suite);
        test.setName("test");
        Map<String, String> parameters = new HashMap<>();
        parameters.put("iterators", "5");
        test.setParameters(parameters);
        test.setXmlClasses(Collections.singletonList(new XmlClass(DataDrivenTestClass.class)));
        testng.setXmlSuites(Collections.singletonList(suite));
        testng.run();
        InputStream isr = new FileInputStream("test-output/testng-failed.xml");
        XmlSuite xmlSuite = new Parser(isr).parseToList().get(0);
        Map<String, String> actualParameters = xmlSuite.getTests().get(0).getLocalParameters();
        Assertions.assertThat(actualParameters).containsAllEntriesOf(parameters);
    }
}