Surefire report is not produced with test name when using JUnit5 and Specs2

[anatolyra]

Hi,

When running tests from terminal via 'mvn clean test' command, the generated surefire report in 'target/surefire-reports/' does not have a tests name. i.e: <testcase name="" classname="samples.MySpecTest" time="0"/> This messes up our TC reporting.

Steps to reproduce

1. Clone my example: Junit5Specs2SurefireTestName
2. Run 'mvn clean test'
3. Open surefire report

When adding this config:

<configuration>
          <statelessTestsetReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5Xml30StatelessReporter">
            <disable>false</disable>
            <version>3.0</version>
            <usePhrasedFileName>false</usePhrasedFileName>
            <usePhrasedTestSuiteClassName>true</usePhrasedTestSuiteClassName>
            <usePhrasedTestCaseClassName>true</usePhrasedTestCaseClassName>
            <usePhrasedTestCaseMethodName>true</usePhrasedTestCaseMethodName>
          </statelessTestsetReporter>
          <consoleOutputReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5ConsoleOutputReporter">
            <disable>false</disable>
            <encoding>UTF-8</encoding>
            <usePhrasedFileName>false</usePhrasedFileName>
          </consoleOutputReporter>
          <statelessTestsetInfoReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5StatelessTestsetInfoReporter">
            <disable>false</disable>
            <usePhrasedFileName>false</usePhrasedFileName>
            <usePhrasedClassNameInRunning>true</usePhrasedClassNameInRunning>
            <usePhrasedClassNameInTestCaseSummary>true</usePhrasedClassNameInTestCaseSummary>
          </statelessTestsetInfoReporter>
        </configuration>

I get this console output:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running null
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.075 s - in null
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------

Notice the 'null' instead of the class name.

And this output in the report: <testcase name="" classname="Single test" time="0.001"/> The test name is in classname but again, nothing in the name attribute.

Context

• maven-surefire-plugin version 3.0.0-M7
• scala-maven-plugin both version 4.4.0 and 4.7.2
• specs2-junit_2.13 version 4.12.1-junit-5 (uses junit-vintage-engine version 5.3.1, also tried with 5.9.0)
• Scala version 2.13.8
• Build Tool/IDE: Maven

I also contacted the surefire team, here's their answer: https://issues.apache.org/jira/browse/SUREFIRE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17622349#comment-17622349

Thanks.

[etorreborre]

Hi. Given the response from the surefire team I don't think that we can really have a proper integration if the plugin is looking for method names which never existed in the first place.

What I would recommend is to add this piece of configuration to your pom file:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.0.0-M7</version>
  <configuration>
    <systemPropertyVariables>
      <junitxml></junitxml>
      <!-- if you want the xml file to be produced in the surefire directory
        <junit.outdir>target/surefire-reports</junit.outdir>
      -->
    </systemPropertyVariables>
  </configuration>

This will produce a correct file, named samples.MySpecTest.xml (instead of TEST-samples.MySpecTest.xml for the surefire one)

[anatolyra]

@etorreborre you mean to replace the surefire generated file with the one generated by Specs2? But this will not work correctly in a mixed project (Scala and Java for example). Java tests will not show up in the output.

[etorreborre]

Doesn't your CI collect all the xml files at the end of the day? Unfortunately I don't have a better solution at the moment.

[anatolyra]

It does. And if I implement this solution I then see the Specs2 tests twice. Once from the faulty Surefire report and the other from the xml produced by Specs2. So, it's a choice between seeing too few tests vs seeing too many tests.

[etorreborre]

I am sorry but I don't know how to make more progress at the moment. You might have to do some xml post processing to get everything right :-(.