ITest is not working with DataProvider

[juherr]

Expected/Actual behavior

test name should be "TestName1" instead of "TestName2[1](TestName2, 2, 2, 4)" .. which is what is showing up in XML reports

Test case sample

public class TestNgProviderExample implements ITest{

    @Test(dataProvider = "summationProvider")
    public void testProvider(String description, int number1, int number2, int sum) {
        Assert.assertEquals(sum, number1 + number2);
    }

    @DataProvider(name = "summationProvider")
    public Object[][] summationData() {
        Object[][] testData = {{"TestName1",1,2,3},{"TestName2",2,2,4}};
        return testData;
    }

    private String reportedTestName = "";

    @BeforeMethod(alwaysRun = true)
    public void testData(Method method, Object[] testData) {
        reportedTestName = testData[0].toString();
    }

    @Override
    public String getTestName() {
        return reportedTestName;
    }
}

From http://stackoverflow.com/q/42726932/4234729

[krmahadevan]

@juherr - So if I understand this problem correctly, what the user here is expecting is to be able to identify one or more parameters from a data provider, which should be used to construct the name ? Is my understanding correct ?

If yes, then we could perhaps consider introducing a new annotation that can be used on the parameters of the data driven test method which would help us identify how to construct the pretty printed name ?

[juherr]

What I understood is: the user expects to be able to customize the test name but TestNG is overriding it.

I didn't test yet and maybe the description of the user is not perfect or even wrong.

But if confirmed, I see 2 problems:

• ITest is overridden by the test params: (TestName2, 2, 2, 4) -> The behavior should be configurable Improvement: We can imagine having an @TestName on a method too which can accept the same parameters than a @BeforeMethod.

• In case of @DataProvider, it seems ITest is checked at the end: TestName2[1] instead of TestName1. I just don't understand why it is TestName2[1] and not TestName2[0].

[krmahadevan]

@juherr - I don't think there's any issues with TestNG. Its more of how the Test reports are being constructed.

Consider the below full fledged example

package com.rationaleemotions.testng;

import org.testng.Assert;
import org.testng.IAlterTestName;
import org.testng.IHookCallBack;
import org.testng.IHookable;
import org.testng.IReporter;
import org.testng.ISuite;
import org.testng.ISuiteResult;
import org.testng.ITestContext;
import org.testng.ITestResult;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
import org.testng.xml.XmlSuite;

import java.lang.annotation.Annotation;
import java.lang.annotation.Retention;
import java.lang.annotation.Target;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

import static java.lang.annotation.ElementType.PARAMETER;

@Listeners(TestNgProviderExample.ChangeTestNameListener.class)
public class TestNgProviderExample implements IHookable {

    @Test(dataProvider = "summationProvider")
    public void testProvider(@Description String description, int number1, int number2, int sum) {
        Assert.assertEquals(sum, number1 + number2);
    }

    @DataProvider(name = "summationProvider")
    public Object[][] summationData() {
        Object[][] testData = {{"TestName1", 1, 2, 3}, {"TestName2", 2, 2, 4}};
        return testData;
    }

    @Override
    public void run(IHookCallBack callBack, ITestResult testResult) {
        Object[] parameters = testResult.getParameters();
        String testName = "";
        List<Integer> indexes = findIndexOfAnnotatedParameter(testResult.getMethod().getConstructorOrMethod()
            .getMethod());
        for (Integer index : indexes) {
            testName += parameters[index].toString();
        }
        if (! testName.trim().isEmpty()) {
            ((IAlterTestName) testResult).setTestName(testName);
        }
        callBack.runTestMethod(testResult);
    }

    private static List<Integer> findIndexOfAnnotatedParameter(Method method) {
        Annotation[][] annotations = method.getParameterAnnotations();
        List<Integer> indexes = new ArrayList<>();
        for (int i = 0; i < annotations.length; i++) {
            if (annotations[i].length > 0) {
                indexes.add(i);
            }
        }
        return indexes;
    }

    @Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
    @Target({PARAMETER})
    @interface Description {}

    public static class ChangeTestNameListener implements IReporter {

        @Override
        public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
            for (ISuite suite : suites) {
                Collection<ISuiteResult> suiteResults = suite.getResults().values();
                for (ISuiteResult suiteResult : suiteResults) {
                    ITestContext context = suiteResult.getTestContext();
                    Set<ITestResult> testResults = new HashSet<>(context.getFailedTests().getAllResults());
                    testResults.addAll(context.getPassedTests().getAllResults());
                    testResults.addAll(context.getSkippedTests().getAllResults());
                    for (ITestResult testResult : testResults) {
                        System.err.println("ITestResult.getName() : " + testResult.getName());
                        System.err.println("ITestResult.getTestName() : " + testResult.getTestName());
                    }
                }
            }
        }
    }

}

And here's the output

===============================================
Default Suite
Total tests run: 2, Failures: 0, Skips: 0
===============================================

ITestResult.getName() : TestName2
ITestResult.getTestName() : null
ITestResult.getName() : TestName1
ITestResult.getTestName() : null

As you can see, getName() works as intended and it retains the value that was passed to it via the IHookable implementation

I think the user's experience is more to do with how the individual reporters are built (including the TestNG built-in default reporters) and the IntelliJ console reporter.

For e.g., what I noticed was that the IntelliJ console reporter (the one that lists the test results and their names etc.,), it tries to first fetch a non empty string from ITest implementation and if it doesn't find it, it resorts to printing the method name along with all the parameters.

So perhaps we need to introduce a new method from within ITestResult (by maybe having it extend another new interface) which when invoked would return a pretty formatted name for the ITestResult and which internally considers the following

• ITest implementation
• testName attribute of @Test annotation
• User's ability to override the name via the org.testng.IAlterTestName#setTestName
• User's ability to set a custom value for the pretty printed name (which will override all the calculations that TestNG does to figure out the name)

And maybe have this rolled out in all our default reports atleast. (Not sure how it would be reflected in IntelliJ TestNG plugin though)

[juherr]

Ok, I saw the answer on StackOverflow and if I understand well your own answer, the "issue" is in the [idea] reporter.

Thanks for the clarification :)

[wgv-ebishop]

Ok, I know this is an old thread, but I don't want to use the @Factory way of solving this problem. I really just want to avoid TestNG adding the [invocation#] to my custom method name.

Is it possible?

[juherr]

@wgv-ebishop As shown by @krmahadevan in its sample. TestNG is working as expected without the need of @Factory. The issue is more in the tool you use to display the test result.

[wgv-ebishop]

This report is being fed into Azure Devops for results tracking. ADO will consume a "JUnit" format. So with this solution, I'd just create the the entire report. I was hoping I could just do something like override a getTestName method on some default class. I can do this by extending JUnitReportReporter, but then ADO just ignores the created report, and used the default JUnitReportReporter. So I may not have another option. I'm not sure the ADO folks won't ignore a custom solution like the one above. But I'll give it a try. Thanks for the reply.

[juherr]

Names in the JUnit report are supposed to be the good ones.

If not, please open a new issue with a way to reproduce the issue.