LegacyXmlReportGeneratingListener should create a Test-nnn.xml fiel for each class

[mseele]

We are migrating to JUnit 5 (from JUnit 4) and using the console launcher in combination with ant like described in https://github.com/junit-team/junit5-samples/blob/master/junit5-jupiter-starter-ant/build.xml

<java classpathref="test.classpath" classname="org.junit.platform.console.ConsoleLauncher" fork="true">
    <arg value="--scan-classpath"/>
    <arg line="--reports-dir build/test-report"/>
</java>

We cannot use junitlauncher-task because the output supports no @ParameterizedTest and so on.

After running the tests we generate a junitreport (with junitreport ant task) for the test run and a unitth report (http://junitth.sourceforge.net/) for the last 100 test runs.

The problem is that the LegacyXmlReportGeneratingListener generates one BIG TEST-nnn.xml for a few hundred test cases. This results in a junitreport without any classes & packages. Just one big list of all tests. The "OLD" JUnit 4 Ant Task generated a TEST-nnn.xml for each test class. With this, junitreport & unitth can create readable reports. The report generated by LegacyXmlReportGeneratingListener is not readable and useless because it exists of one big "TEST" (it even looses the test class names).

Steps to reproduce

Run

<java classpathref="test.classpath" classname="org.junit.platform.console.ConsoleLauncher" fork="true">
    <arg value="--scan-classpath"/>
    <arg line="--reports-dir build/test-report"/>
</java>
<junitreport todir="build/test-report" tofile="report.xml">
    <fileset dir="build/test-report">
        <include name="*.xml" />
    </fileset>
    <report format="frames" todir="build/test-report" />
</junitreport>

with multiple test classes. You just get one big TEST-junit-jupiter.xml instead of a TEST-nnn.xml for each class.

Context

• Used versions (Jupiter/Vintage/Platform): 5.5.1
• Build Tool/IDE: Ant / ConsoleLauncher

[mseele]

My workaround right now is to replace org.junit.platform.reporting.legacy.xml.LegacyXmlReportGeneratingListener with this method replacement:

private void writeXmlReportInCaseOfRoot(TestIdentifier testIdentifier) {
    if (!isRoot(testIdentifier)) {
        UniqueId uniqueId = UniqueId.parse(testIdentifier.getUniqueId());
        if ("class".equals(uniqueId.getLastSegment().getType())) { //$NON-NLS-1$
            writeXmlReportSafely(testIdentifier, testIdentifier.getLegacyReportingName());
        }
    }
}

Which is an ugly workaround, btw.

[mseele]

One more thing: org.junit.platform.reporting.legacy.xml.XmlReportWriter.writeSuiteAttributes(TestIdentifier, List, NumberFormat, XMLStreamWriter) needs to use testIdentifier.getLegacyReportingName() instead of testIdentifier.getDisplayName() to makes it similar to the old junit4 report.

old code: writer.writeAttribute("name", testIdentifier.getDisplayName());

new code: writer.writeAttribute("name", testIdentifier.getLegacyReportingName());

[marcphilipp]

The reason there's a report per test engine is that the JUnit Platform does not require test engines to use classes. However, the resulting XML files contain the class name in the classname attribute of each testcase element. Could you adjust downstream tooling to read it from there or split the XML file using an XSLT transform or sth. similar?

Thinking out loud, I think we could add a reporting granularity level option to the console launcher with 0 being the current level of engines, 1 being their immediate children and so forth. However, since we have a new reporting format in the making (see #373), it doesn't feel like the right time to do so.

One more thing: org.junit.platform.reporting.legacy.xml.XmlReportWriter.writeSuiteAttributes(TestIdentifier, List, NumberFormat, XMLStreamWriter) needs to use testIdentifier.getLegacyReportingName() instead of testIdentifier.getDisplayName() to makes it similar to the old junit4 report.

I assume you mean if we changed the granularity it would start making a difference, right? Currently it's only used to output the test engine's display name.

[marcphilipp]

Tentatively slated for 5.6 Backlog for team discussion.

[mseele]

Sounds like a good idea to make it configurable via console launcher option. That's what I was looking for first.

I'm just saying that the "data" that is generated by the LegacyXmlReportGeneratingListener is not 100% compatible with what we get when we use JUnit4, and thats a stopper for us to migrate to JUnit5.

Unfortunately, the JUnit5 ant task JUnitLauncher does not support @ParameterizedTest and has some other problems, too (e.g. https://bz.apache.org/bugzilla/show_bug.cgi?id=63680 / https://bz.apache.org/bugzilla/show_bug.cgi?id=63446). So this is also no way to go.

I follow the conversation about a new reporting format and would be happy about a universal format. But what is also need IMHO: a good (html) reporting tool that works with hundreds of runs of several thousand tests each run. Currently only JUnit Report -> UnitTH works for us. And UnitTH has not been further developed for years. Is there possibly something better that we don't know yet?

[kriegfrj]

PR #1981 could be used as the basis of a solution to this (see https://github.com/junit-team/junit5/issues/1980#issuecomment-524556192 as an example).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. Thank you for your contribution.

[ben-spiller]

+1 for this, please

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. Thank you for your contribution.