Suite with test with the same name alias each other result

[fabio-boldrini]

We have kind this suite. There are two test with the same name, the first fail and the second not fail.

When we ran this test suite from eclipse testng plugin, it show two test, the first one failing and the second one passed.

But in the IReport passed as report listener only the last one win (the passed).

I dont know if it is a bug (it's weird to have tests with the same name...) but the eclipse plugin do it right (cause i think it's hocked at single test execution for real-time plugin gui update), so i ask if it was intended or not.

Here a test to show that:

import java.util.Arrays;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;

import org.testng.Assert;
import org.testng.IReporter;
import org.testng.IResultMap;
import org.testng.ISuite;
import org.testng.ISuiteResult;
import org.testng.ITestContext;
import org.testng.TestNG;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlSuite;
import org.testng.xml.XmlTest;

public class TestNgOnDuplicateTestName {

    public static class ReportThatMustFail implements IReporter {
        @Override
        public void generateReport(List<XmlSuite> arg0, List<ISuite> suites, String arg2) {
            ISuite suite = suites.iterator().next();
            Map<String, ISuiteResult> resultMap = suite.getResults();
            ISuiteResult result = resultMap.entrySet().iterator().next().getValue();
            ITestContext testContext = result.getTestContext();

            IResultMap failedTests = testContext.getFailedTests();
            IResultMap passedTests = testContext.getPassedTests();

            Assert.assertEquals(failedTests.size(), 1, "must have one failed test");
            Assert.assertEquals(passedTests.size(), 1, "must have one passed test");
        }
    }

    public static class MyTest {
        @Parameters({"mustFail"})
        @Test
        public void test(String mustFail) {
            boolean t = Boolean.parseBoolean(mustFail);
            Assert.assertFalse(t);
        }
    }

    @Test
    public void testOnDuplicateTestNameFirstFailSecondSucceded() {
        TestNG testng = new TestNG();
        testng.setVerbose(0);

        XmlSuite xmlSuite = getXmlSuite();

        XmlTest testThatFail = createMyTest(xmlSuite, true);
        XmlTest testThatNotFail = createMyTest(xmlSuite, false);

        xmlSuite.setTests(Arrays.asList(testThatFail, testThatNotFail));

        testng.setXmlSuites(Arrays.asList(xmlSuite));

        testng.run();
    }

    @Test
    public void testOnDuplicateTestNameFirstSuccededSecondFail() {
        TestNG testng = new TestNG();
        testng.setVerbose(0);

        XmlSuite xmlSuite = getXmlSuite();

        XmlTest testThatFail = createMyTest(xmlSuite, true);
        XmlTest testThatNotFail = createMyTest(xmlSuite, false);

        xmlSuite.setTests(Arrays.asList(testThatNotFail, testThatFail));

        testng.setXmlSuites(Arrays.asList(xmlSuite));

        testng.run();
    }

    private XmlSuite getXmlSuite() {
        XmlSuite xmlSuite = new XmlSuite();
        xmlSuite.setName("TEST_SUITE_THAT_MUST_FAILE");
        xmlSuite.setListeners(Arrays.asList("TestNgOnDuplicateTestName$ReportThatMustFail"));
        return xmlSuite;
    }

    private XmlTest createMyTest(XmlSuite xmlSuite, boolean mustFail) {
        XmlTest xmlTestThatFail = new XmlTest(xmlSuite);

        xmlTestThatFail.setName("TEST_SAME_NAME");

        xmlTestThatFail.setXmlClasses(Arrays.asList(new XmlClass(MyTest.class)));

        xmlTestThatFail.addParameter("mustFail", mustFail ? "true" : "false");

        return xmlTestThatFail;
    }
}

[cbeust]

TestNG expects these names to be unique across a tag, so you should either use a different name or have one of your test classes implement org.testng.ITest to define a different name.