search

testng-team / testng

TestNG testing framework
https://testng.org
Apache License 2.0
1.98k stars 1.02k forks source link

NPE in JUnitXMLReporter #993

Closed juherr closed 5 years ago

juherr commented 8 years ago 
Failed to execute goal   org.apache.maven.plugins:maven-failsafe-plugin:2.19.1:integration-test (run-tests) on project XXXX: Execution run-tests of goal org.apache.maven.plugins:maven-failsafe-plugin:2.19.1:integration-test failed: There was an error in the forked process
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:459)
at java.util.Properties.setProperty(Properties.java:166)
at org.testng.reporters.JUnitXMLReporter.createElement(JUnitXMLReporter.java:207)
at org.testng.reporters.JUnitXMLReporter.createElementFromTestResults(JUnitXMLReporter.java:187)
at org.testng.reporters.JUnitXMLReporter.generateReport(JUnitXMLReporter.java:178)
at org.testng.reporters.JUnitXMLReporter.onFinish(JUnitXMLReporter.java:118)
at org.testng.TestRunner.fireEvent(TestRunner.java:1239)
at org.testng.TestRunner.afterRun(TestRunner.java:1030)
at org.testng.TestRunner.run(TestRunner.java:636)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
at org.testng.SuiteRunner.run(SuiteRunner.java:268)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1169)
at org.testng.TestNG.run(TestNG.java:1064)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:132)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:147)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
 at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
 [Help 1]

From http://stackoverflow.com/q/35605443/4234729

krmahadevan commented 7 years ago

@juherr - The SO link you provided doesn't seem to be correct. Its taking me to https://stackoverflow.com/questions/218384/what-is-a-nullpointerexception-and-how-do-i-fix-it

Was wondering if you could add a bit more details on how to reproduce this problem.

juherr commented 7 years ago

It is https://stackoverflow.com/questions/35605443/testng-nullpointerexception-in-junitxmlreporter I think we won't be able to reproduce the issue but the issue shows it is possible to set null somewhere it should not be. So, we should check the null value somewhere in the stacktrace even if the issue may come from surefire.

krmahadevan commented 7 years ago

Interesting.. Everytime I try to load https://stackoverflow.com/questions/35605443/testng-nullpointerexception-in-junitxmlreporter , I end up in https://stackoverflow.com/questions/218384/what-is-a-nullpointerexception-and-how-do-i-fix-it

The stack trace doesn't correlate with the current codebase.. So its essentially going to be shooting in the dark.

juherr commented 7 years ago

Ok, so we can close the issue.

sirisov commented 5 years ago

I have this exception right now on testng 6.13.1 :

java.lang.NullPointerException
    at java.util.Hashtable.put(Hashtable.java:460)
    at java.util.Properties.setProperty(Properties.java:166)
    at org.testng.reporters.JUnitXMLReporter.createElement(JUnitXMLReporter.java:227)
    at org.testng.reporters.JUnitXMLReporter.createElementFromTestResults(JUnitXMLReporter.java:187)
    at org.testng.reporters.JUnitXMLReporter.generateReport(JUnitXMLReporter.java:171)
    at org.testng.reporters.JUnitXMLReporter.onFinish(JUnitXMLReporter.java:112)
    at org.testng.TestRunner.fireEvent(TestRunner.java:772)
    at org.testng.TestRunner.afterRun(TestRunner.java:741)
    at org.testng.TestRunner.run(TestRunner.java:509)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
    at org.testng.SuiteRunner.run(SuiteRunner.java:364)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1187)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1116)
    at org.testng.TestNG.runSuites(TestNG.java:1028)
    at org.testng.TestNG.run(TestNG.java:996)
    at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:132)
    at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:193)
    at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:94)
    at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:147)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)

This is happening in case of having dataprovider and class with ITest interface, like this:

public class Test implements ITest {

  private String testName;

  @BeforeMethod
  public void setTestName(Method method, Object[] testData) {
    testName = method.getName() + Arrays.toString(testData);
  }

  @Test(dataProvider = "DP", dataProviderClass = DataProviders.class)
  public void Test_001(String param) {
    ...
  }

  @Override
  public String getTestName() {
    return testName;
  }

}

This is not exact code for reproducing. At least I can't make short reproducible code. Our real class have custom listeners and reporters. But exception disappears in two cases: 1) Move @Test annotation with dataprovider at class level (real test class have several methods and all requires parameters - this is why I don't see that exception until I wanted to run only single method and move annotation from class level to method level 2) Initialize testName with some initial value - this is the reason of exception - reporter tries to get name (before running the test?) and gets null

krmahadevan commented 5 years ago

@sirisov - Thanks for providing that additional context. May I please request you to retry using TestNG 7.0.0-beta1 (latest released version) and if the problem persists, also try using TestNG 7.0.0-SNAPSHOT the upcoming release's snapshot and let us know if the problem persists ?

juherr commented 5 years ago

I reopen the issue. Please provide a runnable project that will allow reproducing the issue easily.

krmahadevan commented 5 years ago

@sirisov - This was the only sample that I could think of, which can simulate the problem that you are experiencing.

import java.lang.reflect.Method;
import org.testng.ITest;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class SampleTestClass implements ITest {

  private String testName;

  @BeforeMethod
  public void setTestName(Method method, Object[] testData) {
    testName = (String) testData[0];
  }

  @Test(dataProvider = "DP")
  public void Test_001(String param) {}

  @DataProvider(name = "DP")
  public Object[][] getData() {
    return new Object[][] {{null}};
  }

  @Override
  public String getTestName() {
    return testName;
  }
}
java.lang.NullPointerException
    at java.util.Hashtable.put(Hashtable.java:459)
    at java.util.Properties.setProperty(Properties.java:166)
    at org.testng.reporters.JUnitXMLReporter.createElement(JUnitXMLReporter.java:227)
    at org.testng.reporters.JUnitXMLReporter.createElementFromTestResults(JUnitXMLReporter.java:187)
    at org.testng.reporters.JUnitXMLReporter.generateReport(JUnitXMLReporter.java:171)
    at org.testng.reporters.JUnitXMLReporter.onFinish(JUnitXMLReporter.java:112)
    at org.testng.TestRunner.fireEvent(TestRunner.java:772)
    at org.testng.TestRunner.afterRun(TestRunner.java:741)
    at org.testng.TestRunner.run(TestRunner.java:509)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
    at org.testng.SuiteRunner.run(SuiteRunner.java:364)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
    at org.testng.TestNG.runSuites(TestNG.java:1049)
    at org.testng.TestNG.run(TestNG.java:1017)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
juherr commented 5 years ago

Good catch @krmahadevan 👍

But we should not have an NPE and manage the case (or fail before with an explicit message).

krmahadevan commented 5 years ago

I was thinking we do one of the two

  1. We stringify null into "null"
  2. If we find the ITest.getTestName() null, we fall back to the method name ( but this won't tell the user what really went wrong )

@juherr your thoughts ?

juherr commented 5 years ago

I prefer the fallback solution. And you can log a warning too.

Or we can consider that the name is not nullable and you can throw an exception if null because the user made an error he can fix.