search

testng-team / testng

TestNG testing framework
https://testng.org
Apache License 2.0
1.99k stars 1.02k forks source link

NullPointer on TestHTMLReporter.generateLog when i run test in parrallel #2221

Closed mblanloeil closed 4 years ago

mblanloeil commented 4 years ago

TestNG 7.1.0

Note: only the latest version is supported

Expected behavior

I want run all test in parrallel (classes)

Actual behavior

All tests are succedeed, but maven goal crash on reporting : 

> [INFO] Tests run: 1333, Failures: 0, Errors: 0, Skipped: 0

> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test (default-test) on project core: There are test failures.
[ERROR] 
[ERROR] Please refer to /mnt/kosmos-produit/koreparent/core/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] ExecutionException There was an error in the forked processjava.lang.NullPointerException
[ERROR]         at java.io.FileOutputStream.<init>(FileOutputStream.java:203)
[ERROR]         at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
[ERROR]         at org.testng.internal.Utils.openWriter(Utils.java:194)
[ERROR]         at org.testng.internal.Utils.openWriter(Utils.java:182)
[ERROR]         at org.testng.reporters.TestHTMLReporter.generateLog(TestHTMLReporter.java:276)
[ERROR]         at org.testng.reporters.TestHTMLReporter.onFinish(TestHTMLReporter.java:38)
[ERROR]         at org.testng.TestRunner.fireEvent(TestRunner.java:895)
[ERROR]         at org.testng.TestRunner.afterRun(TestRunner.java:859)
[ERROR]         at org.testng.TestRunner.run(TestRunner.java:590)
[ERROR]         at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
[ERROR]         at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
[ERROR]         at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
[ERROR]         at org.testng.SuiteRunner.run(SuiteRunner.java:286)
[ERROR]         at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
[ERROR]         at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
[ERROR]         at org.testng.TestNG.runSuitesSequentially(TestNG.java:1187)
[ERROR]         at org.testng.TestNG.runSuitesLocally(TestNG.java:1109)
[ERROR]         at org.testng.TestNG.runSuites(TestNG.java:1039)
[ERROR]         at org.testng.TestNG.run(TestNG.java:1007)
[ERROR]         at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
[ERROR]         at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
[ERROR]         at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123)
[ERROR]         at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90)
[ERROR]         at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
[ERROR]         at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
[ERROR]         at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
[ERROR]         at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
[ERROR]         at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
[ERROR]

Is the issue reproductible on runner?

Test case sample

Just run mvn test

I use maven-surefire-plugin for run test : 

<dependency>
      <groupId>org.testng</groupId>
      <artifactId>testng</artifactId>
      <version>7.1.0</version>
    </dependency>
    <dependency>
      <groupId>org.jmockit</groupId>
      <artifactId>jmockit</artifactId>
      <version>1.49</version>
    </dependency>
<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.0.0-M4</version>
        <configuration>
          <!-- Sets the VM argument line used when unit tests are run. -->
          <argLine>
            -javaagent:${settings.localRepository}/org/jmockit/jmockit/${jmockit.version}/jmockit-${jmockit.version}.jar
          </argLine>
          <!-- Skips unit tests if the value of skip.unit.tests property is true -->
          <skipTests>${skip.unit.tests}</skipTests>
          <forkCount>4</forkCount>
          <threadCount>2</threadCount>
          <reuseForks>true</reuseForks>
          <parallel>classes</parallel>
          <!-- Excludes integration tests when unit tests are run. -->
          <excludes>
            <exclude>**/IT*.java</exclude>
          </excludes>
        </configuration>
      </plugin>
krmahadevan commented 4 years ago

Not able to reproduce this issue using 7.1.0. I used the attached sample project and it works fine. github-2221.zip

The stack trace seems to suggest that the file name somehow came in as null, but looking at the code path, I cant seem to find a way that it could be possible.

So I would need a reproducible sample that can be used to recreate this issue.

Please comment once you have one, and we can re-open this issue.

mblanloeil commented 4 years ago

Hello,

I finally reproduce this issue when I have a class test who mocked File class. You can use my sample (github-2221.zip) and launch mvn command : mvn -Dtest=KungFuPandaTest test github-2221.zip

krmahadevan commented 4 years ago

@mblanloeil - I took a closer look at your issue. The root cause of the problem is because you are mocking File object and the mocked object is what is getting exposed to TestNG as well (TestNG internally also uses the File object for report generation).

I am not well versed with Mocking File Objects, but my googling suggests that its a bad idea to mock a file object.

So at this juncture there's really not much that can be done from TestNG perspective since the underlying exception is arising from the JDK apis due to the interference of mocks.

So we might have to close this issue with no resolution from TestNG side.

@juherr - Do you have any suggestions on this ?

juherr commented 4 years ago

Default reporters can be disabled and you can use custom one which doesn't use File. It could work.

As pointed by @krmahadevan's link, I'm not sure using File is a good idea and maybe shows a design issue somewhere. You should check if it is possible to modify the internal usage of File by something more configurable (like In/OutputStream).

krmahadevan commented 4 years ago

@juherr - Should we close this issue, since there's not much that can be done from TestNG side.