Surefire Discovers one testPlan per testClass, leading to generated reports overwriting each other

[Michihalo]

When tests are executed via surefire, they are discovered on a per test-class basis. For each test-class a separate testPlan is created. The EnhancedLegacyXmlReportGeneratingListener creates one report per testPlan. It takes the name of the root as the name for the report, which is the engine name (junit-jupiter) in all of those cases. Surefire itself reports all classes in a separate file.

The problem could possibly be here:

I am not a surefire pro, so it is possible the surefire configuration is not correct. Nevertheless, surefire itself is reporting per class and this listener is not. I tried to debug and had the feeling that no special testPlan was created fo the EnhancedLegacyXmlReportGeneratingListener.

[bitcoder]

Thanks for reporting.. will need to have a deeper look at this

[bitcoder]

Can you provide some basic sample project code that shows this behavior as I was unable to replicate it?

[bitcoder]

I was able to replicate it, no need for a sample code

[bitcoder]

I think this is related to the version of surefire being used. please try a recent version (e.g. 3.2.5); that worked for me. I was using surefire 2.22.2 and that version didn't work well with JUnit5, leading to to one testplan for test class. Here's the pom.xml I've used on my sample project:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.idera.xray.tutorials</groupId>
    <artifactId>java-junit5-selenium</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties> 
        <junit-jupiter.version>5.10.2</junit-jupiter.version>
        <junit-platform.version>1.10.2</junit-platform.version>    
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.junit</groupId>
                <artifactId>junit-bom</artifactId>
                <version>${junit-jupiter.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>app.getxray</groupId>
            <artifactId>xray-junit-extensions</artifactId>
            <version>0.7.1</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>app.getxray</groupId>
            <artifactId>xray-maven-plugin</artifactId>
            <version>0.7.2</version>
        </dependency>

        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>4.1.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <debug>true</debug>
                    <source>1.8</source>
                    <target>1.8</target>
                    <compilerArgs>
                        <!-- this flag is required in order to obtain the name of parameters in test methods -->
                        <arg>-parameters</arg>
                    </compilerArgs>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.2.5</version>
                <configuration>
                    <testFailureIgnore>true</testFailureIgnore>
                    <disableXmlReport>true</disableXmlReport>
                    <useFile>false</useFile>
                </configuration>
            </plugin>

            <plugin>
                <groupId>app.getxray</groupId>
                <artifactId>xray-maven-plugin</artifactId>
                <version>0.7.2</version>
                <configuration>
            <cloud>false</cloud>
                    <projectKey>XT</projectKey>
                    <reportFormat>junit</reportFormat>
                    <reportFile>reports/TEST-junit-jupiter.xml</reportFile>
                </configuration>
            </plugin>

        </plugins>
    </build>

    <reporting>
        <plugins>
            <plugin>
                <artifactId>maven-surefire-report-plugin</artifactId>
            </plugin>
        </plugins>
    </reporting>

</project>

[Michihalo]

Thanks for the quick investigation. I can verify that it works as you described with version 3.2.5

[bitcoder]

I'll add a reference in the docs to clarify this and then close this issue. Thanks!

[bitcoder]

FAQ section updated on the README file.