Test name issue in Reporter with DataProvider

[juherr]

From the mailing list: https://groups.google.com/forum/#!topic/testng-users/p0QRhby9O4o

Sample from @krmahadevan

public class Example {
    public static TestNG create() {
        TestNG result = new TestNG();
        result.setUseDefaultListeners(false);
        result.setVerbose(0);
        return result;
    }

    public static TestNG create(Class<?>... testClasses) {
        TestNG result = create();
        result.setTestClasses(testClasses);
        return result;
    }

    @Test
    public void complexITestTest() {
        TestNG tng = create(ITestSample.class);
        tng.setDataProviderThreadCount(2);
        TestListenerAdapter adapter = new TestListenerAdapter();
        tng.addListener(adapter);
        LocalReporterListenerAdaptor reporter = new LocalReporterListenerAdaptor();
        tng.addListener(reporter);
        ExecutorService service = Executors.newFixedThreadPool(1);
        service.submit(new Runnable() {
            @Override
            public void run() {
                tng.run();
            }
        });

        Assert.assertTrue(adapter.getFailedTests().isEmpty());
        Assert.assertTrue(adapter.getSkippedTests().isEmpty());
        Assert.assertEquals(adapter.getPassedTests().size(), 5);
        List<String> testNames = new ArrayList<>(Arrays.asList("test1", "test2", "test3", "test4", "test5"));
        for (ITestResult testResult : reporter.getResults()) {
            Assert.assertTrue(testNames.remove(testResult.getName()),
                "Duplicate test names " + getNames(adapter.getPassedTests()));
        }
        Assert.assertEquals(testNames, Collections.emptyList());
    }

    private static List<String> getNames(List<ITestResult> results) {
        List<String> names = new ArrayList<>(results.size());
        for (ITestResult result : results) {
            names.add(result.getName());
        }
        return names;
    }

    public static class LocalReporterListenerAdaptor implements IReporter {
        private Set<ITestResult> results;

        @Override
        public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
            ITestContext testContext = suites.get(0).getResults().values().iterator().next().getTestContext();
            IResultMap resultMap = testContext.getPassedTests();
            results = resultMap.getAllResults();
        }

        public Set<ITestResult> getResults() {
            return results;
        }
    }

    public static class ITestSample implements ITest {
        public ThreadLocal<String> testName = new ThreadLocal<>();

        @DataProvider (name = "dp", parallel = true)
        public Object[][] getTests() {
            return new Object[][] {new Object[] {"test1"},
                new Object[] {"test2"},
                new Object[] {"test3"},
                new Object[] {"test4"},
                new Object[] {"test5"}};

        }

        @Test (dataProvider = "dp")
        public void run(String testName) {
            Assert.assertEquals(testName, this.testName.get());
        }

        @BeforeMethod
        public void init(Object[] testArgs) {
            testName.set((String) testArgs[0]);
        }

        @AfterMethod
        public void tearDown(Object[] testArgs) {
            Assert.assertEquals((String) testArgs[0], this.testName.get());
        }

        @Override
        public String getTestName() {
            return testName.get();
        }
    }

}

[krmahadevan]

@Julien, Do you think that this is something that needs to be fixed from within TestNG ? IMO this is something that can be handled by a test repository specific custom reporter because TestNG still gives that mechanism.

Also please let me know if we are to be fixing this at the TestNG provided default reporters.

Thanks & Regards Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!" My Scribblings @ http://wakened-cognition.blogspot.com/ My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Wed, Jul 13, 2016 at 12:35 AM, Julien Herr notifications@github.com wrote:

From the mailing list: https://groups.google.com/forum/#!topic/testng-users/p0QRhby9O4o

Sample from @krmahadevan https://github.com/krmahadevan

public class Example { public static TestNG create() { TestNG result = new TestNG(); result.setUseDefaultListeners(false); result.setVerbose(0); return result; }

public static TestNG create(Class<?>... testClasses) {
    TestNG result = create();
    result.setTestClasses(testClasses);
    return result;
}

@Test
public void complexITestTest() {
    TestNG tng = create(ITestSample.class);
    tng.setDataProviderThreadCount(2);
    TestListenerAdapter adapter = new TestListenerAdapter();
    tng.addListener(adapter);
    LocalReporterListenerAdaptor reporter = new LocalReporterListenerAdaptor();
    tng.addListener(reporter);
    ExecutorService service = Executors.newFixedThreadPool(1);
    service.submit(new Runnable() {
        @Override
        public void run() {
            tng.run();
        }
    });

    Assert.assertTrue(adapter.getFailedTests().isEmpty());
    Assert.assertTrue(adapter.getSkippedTests().isEmpty());
    Assert.assertEquals(adapter.getPassedTests().size(), 5);
    List<String> testNames = new ArrayList<>(Arrays.asList("test1", "test2", "test3", "test4", "test5"));
    for (ITestResult testResult : reporter.getResults()) {
        Assert.assertTrue(testNames.remove(testResult.getName()),
            "Duplicate test names " + getNames(adapter.getPassedTests()));
    }
    Assert.assertEquals(testNames, Collections.emptyList());
}

private static List<String> getNames(List<ITestResult> results) {
    List<String> names = new ArrayList<>(results.size());
    for (ITestResult result : results) {
        names.add(result.getName());
    }
    return names;
}

public static class LocalReporterListenerAdaptor implements IReporter {
    private Set<ITestResult> results;

    @Override
    public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
        ITestContext testContext = suites.get(0).getResults().values().iterator().next().getTestContext();
        IResultMap resultMap = testContext.getPassedTests();
        results = resultMap.getAllResults();
    }

    public Set<ITestResult> getResults() {
        return results;
    }
}

public static class ITestSample implements ITest {
    public ThreadLocal<String> testName = new ThreadLocal<>();

    @DataProvider (name = "dp", parallel = true)
    public Object[][] getTests() {
        return new Object[][] {new Object[] {"test1"},
            new Object[] {"test2"},
            new Object[] {"test3"},
            new Object[] {"test4"},
            new Object[] {"test5"}};

    }

    @Test (dataProvider = "dp")
    public void run(String testName) {
        Assert.assertEquals(testName, this.testName.get());
    }

    @BeforeMethod
    public void init(Object[] testArgs) {
        testName.set((String) testArgs[0]);
    }

    @AfterMethod
    public void tearDown(Object[] testArgs) {
        Assert.assertEquals((String) testArgs[0], this.testName.get());
    }

    @Override
    public String getTestName() {
        return testName.get();
    }
}

}

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cbeust/testng/issues/1089, or mute the thread https://github.com/notifications/unsubscribe/AA3hzaQLp07dmrfsahc2zaLsawGx4KyDks5qU-VsgaJpZM4JKt8n .

[julien]

no idea

[juherr]

No more idea for the moment :) I just created the issue to keep a trace.

@julien :+1:

[athrunsun]

@juherr @krmahadevan What if I want to change testName when I run the test method, when I am using @DataProvider and my test data is fetched somewhere else?

Sample code:

public class ITestSample implements ITest {
    public ThreadLocal<String> testName = new ThreadLocal.withInitial("Initial test name");

    @DataProvider (name = "dp", parallel = true)
    public Object[][] getTests() {
        Object[][] testData = getTestDataFromSomewhereElse();
        return testData;
    }

    @Test(dataProvider = "dp")
    public void runTest(String testName) {
        this.testName.set(testName);
        Assert.assertEquals(testName, this.testName.get());
    }

    @Override
    public String getTestName() {
        return testName.get();
    }
}

This sample will print all test names as "Initial test name".

I assue TestNG's ITest.getTestName() requires me to determine testName when TestNG initialize my test class right?

BTW, is there an easy way to change testName in TestNG's listeners? When using @DataProvider, all tests have the same test method name printed on TestNG's test report.

[juherr]

@athrunsun As I know, it is not possible to do it easily.

It may work with a factory:

public class ITestSample implements ITest {
    public final ThreadLocal<String> testName = new ThreadLocal.withInitial("Initial test name");

    @DataProvider (name = "dp", parallel = true)
    public static Object[][] getTests() {
        Object[][] testData = getTestDataFromSomewhereElse();
        return testData;
    }

    @Factory(dataProvider = "dp")
    public ITestSample(String testName) {
        this.testName.set(testName);
    }

    @Override
    public String getTestName() {
        return testName.get();
    }

    @Test
    public void runTest() {
    }
}

[athrunsun]

@juherr TestNG is using reflection to invoke dataprovider method to create ITestSample instances, which means I have to make my program's context ready when it calls getTestDataFromSomewhereElse();, so that TestNG can determine the test name. And that leads to many destructive changes to my program.

Just as I said in my previous comment, I assume ITest.getTestName() is designed to determine custom test name during test class(es) initialization phase (when using @Factory), not test execution phase. But actually many things won't happen until test execution phase (inside getTestDataFromSomewhereElse()).