Wrong name and class name in XML report for parameterized tests

[bbilger]

A test like this

package com.example;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
public class SomeTest {
   @ParameterizedTest
   @ValueSource(strings = { "val0", "val1" })
   void testSomething(String someValue) {
   }
}

results in an XML report (TEST-junit-jupiter.xml) like this:

...
<testcase name="testSomething(String)" classname="testSomething(String)" time="0">...</testcase>
<testcase name="testSomething(String)" classname="testSomething(String)" time="0">...</testcase>
...

A similar test in JUnit 4 (@RunWith(Parameterized.class)), however, produces something like this:

...
<testcase name="testSomething[0]" classname="com.example.SomeTest" time="0.0"/>
<testcase name="testSomething[1]" classname="com.example.SomeTest" time="0.0"/>
...

My "problem" in particular or rather observation is that after migrating to JUnit 5 the number of tests reported in sonar dropped and the tests only using @ParameterizedTest no longer show up. Since as far as I know, sonar parses those XML reports, it's the only explanation I have for that drop. This for sure is not limited to sonar but all tools parsing those XML reports.

So it obviously has no high priority, at all but it would be great if you could fix this at some point.

In case it is relevant: I am using gradle.

buildscript {
  repositories {
    mavenCentral()
  }
  dependencies {
    classpath 'org.junit.platform:junit-platform-gradle-plugin:1.0.2'
  }
}
repositories {
  mavenCentral()
}

apply plugin: 'eclipse'
apply plugin: 'java'
apply plugin: 'org.junit.platform.gradle.plugin'

dependencies {
  testCompile(
    'org.junit.jupiter:junit-jupiter-api:5.0.2',
    'org.junit.jupiter:junit-jupiter-params:5.0.2'
  )
  testRuntime(
    'org.junit.jupiter:junit-jupiter-engine:5.0.2',
    'org.junit.platform:junit-platform-launcher:1.0.2'
  )
}

Deliverables

• [x] Ensure that attribute values (e.g., name and classname) in XML reports make sense for the following use cases.
  • [x] repeated tests
  • [x] parameterized tests
  • [x] dynamic containers
  • [x] dynamic tests

[sbrannen]

Although we will eventually migrate to a new report format that understands the JUnit Platform, I think we might be able to improve this reporting scenario in the interim.

I am therefore slating this for 5.1 so that we take a look at it.

[sbrannen]

I've labeled this as a bug since the classname in such cases is obviously incorrect.

[sbrannen]

I've also added a Deliverables section to this issue.

[obfischer]

I stumpled on the same problem. Also for repeated tests (@RepeatedTest) is the method name instead of the classname given.

[obfischer]

The Junit plugin of jQAssistant uses the written report to link test results and Java methods and classes. There relations created by allow the user to search for classes and methods with brocken tests.

Therefore it is important for us that the report allows us to identify the method and the class behind a test case.

[m-g-sonar]

Thanks for reporting this. On SonarJava side (Java plugin from SonarQube handling parsing of surefire reports), we are tentatively going to workaround the issue to not suffer too much of the number of test drop. When fixed, it should get back to normal and be handled correctly. Associated ticket in our JIRA: SONARJAVA-2580.

[marcphilipp]

in progress

[marcphilipp]

@bbilger @obfischer @m-g-sonar I've submitted a PR to fix this (see #1236). It would be great if you could take the XML reports from the PR description and check whether they meet your expectations.

[m-g-sonar]

@marcphilipp Manually tested the file(s) you are providing with our latest build of SonarJava, works like a charm. Thanks for the follow up.

[marcphilipp]

@m-g-sonar Thanks for checking!

[obfischer]

@marcphilipp Sorry for the late response. For me it is also ok. I can now compute all values I need.

[marcphilipp]

@obfischer Thanks for letting us know! 👍

[smoyer64]

I suspect those reporting an issue with SonarQube above are seeing the same issue that I am (lowered test counts). It should be noted that test coverage is still correct based on the execution of multiple tests within a parameterized test.

[marcphilipp]

We've just released 5.0.3 which should fix it.

[bitcoder]

Hi guys,

as of 5.2.0, the XML report generates something like this for a parameterized test.

    @DisplayName("Roman numeral")
    @ParameterizedTest(name = "run #{index} with [{arguments}]")
    @ValueSource(strings = { "Hello", "JUnit" })
    void test2(String word) { 
        assert(word.length()>4);
    }

  <testcase name="test2{String}[1]" classname="com.xpand.java.SimpleTest" time="0.001"/>
  <testcase name="test2{String}[2]" classname="com.xpand.java.SimpleTest" time="0.001"/>

My questions are:

1. shouldn't the name attribute of the @ParameterizedTest annotation be mapped into the name of the testcase in the XML report?
2. aren't (or can't) the arguments saved in the testcase element of the XML report? the current notation isn't that helpful because we can't analyze the results and understand to which parameters that entry belongs to if we just look at "test2{String}[1]"... that is being mapped into the "name" attribute; this seems to be possible though in JUnit4 using the runners JUnitParamsRunner or Parameterized.

[marcphilipp]

I absolutely agree that both the display name and the parameters should be visible in the report. However, the XML format does not support it and the current output is carefully curated to be compatible with widely used tools. You're using Maven, right? In that case the report is written by Surefire which does not support display names and parameters in its internal API, either.

What we really need is a new reporting format. There's an issue for that: https://github.com/ota4j-team/opentest4j/issues/9. 😉

[bitcoder]

Yes @marcphilipp, I'm using maven and surefire plugin. It seems that the output generated with JUnit4 generates a "standard" JUnit xml where the "name" attribute contains the parameters as seen below (using two different runners).

  <testcase name="testCompute[0: fib(0)=0]" classname="com.xpand.java.Fibonacci3Test" time="0.002"/>
 <testcase name="personIsAdultInline(17, false) [0]" classname="com.xpand.java.PersonTest" time="0.042"/>

The first example was generated with the "Parameterized" class runner,

package com.xpand.java;

import static org.junit.Assert.assertEquals;

import java.util.Arrays;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class Fibonacci3Test {

    @Parameters(name = "{index}: fib({0})={1}")
    public static Iterable<Object[]> data() {
        return Arrays.asList(new Object[][] { 
                 { 0, 0 }, { 1, 1 }, { 2, 1 }, { 3, 2 }, { 4, 3 }, { 5, 5 }, { 6, 8 }
           });
    }

    private int input;
    private int expected;

    public Fibonacci3Test(int input, int expected) {
        this.input = input;
        this.expected = expected;
    }

    @Test
    public void testCompute() {
        assertEquals(expected, Fibonacci.compute(input));
    }
}

  <testcase name="testCompute[0: fib(0)=0]" classname="com.xpand.java.Fibonacci3Test" time="0.002"/>
  <testcase name="testCompute[1: fib(1)=1]" classname="com.xpand.java.Fibonacci3Test" time="0"/>
  <testcase name="testCompute[2: fib(2)=1]" classname="com.xpand.java.Fibonacci3Test" time="0"/>
  <testcase name="testCompute[3: fib(3)=2]" classname="com.xpand.java.Fibonacci3Test" time="0"/>
  <testcase name="testCompute[4: fib(4)=3]" classname="com.xpand.java.Fibonacci3Test" time="0"/>
  <testcase name="testCompute[5: fib(5)=5]" classname="com.xpand.java.Fibonacci3Test" time="0"/>
  <testcase name="testCompute[6: fib(6)=8]" classname="com.xpand.java.Fibonacci3Test" time="0"/>

The second example code, using JUnitParamsRunner class, is:

package com.xpand.java;

import static org.assertj.core.api.Assertions.assertThat;

import org.junit.Test;
import org.junit.runner.RunWith;

import junitparams.JUnitParamsRunner;
import junitparams.Parameters;

@RunWith(JUnitParamsRunner.class)
public class PersonTest {

    @Test
    @Parameters({ 
            "17, false", 
            "22, true" })
    public void personIsAdultInline(int age, boolean valid) throws Exception {
        assertThat((new Person(age)).isAdult()).isEqualTo(valid);
    }

    @Test
    @Parameters(method = "adultValues")
    public void personIsAdultMethod(int age, boolean valid) throws Exception {
        //assertEquals(valid, new Person(age).isAdult());
        assertThat(new Person(age).isAdult()).isEqualTo(valid);
    }

    private Object[] adultValues() {
        return new Object[]{
                     new Object[]{13, false},
                     new Object[]{17, false},
                     new Object[]{18, true},
                     new Object[]{22, true}
                };
    }
}

  <testcase name="personIsAdultInline(17, false) [0]" classname="com.xpand.java.PersonTest" time="0.042"/>
  <testcase name="personIsAdultInline(22, true) [1]" classname="com.xpand.java.PersonTest" time="0"/>

[bitcoder]

My suggestion would be to provide some interim solution. We're a Test Management vendor for Jira and our addon (Xray) (https://marketplace.atlassian.com/apps/1211769/xray-test-management-for-jira?hosting=server&tab=overview) provides already handling of JUnit XML reports. However, our users have the need of:

• parameterized tests support
• allow to tag Tests with certain labels / Jira issue keus

the problem that we're facing with JUnit is that the XML report does not provide visibility of this, so our users cannot take advantage of this information in our reports because we simply are unable to process them. With TestNG this works fine though, so some users are using it instead. Of course we want to promote also JUnit and I think if we were able to get a way of at least deal with parameterized tests in some way (for example, as I mentioned in the JUnit4 examples above using two different runners). Concernings tags, that's a different discussion but I think they are not viable with the current XML report format right? Well, maybe they could be added as nested elements under the testcase and that wouldn't break anything. Thanks for your help.

[marcphilipp]

Understood. In this case Maven Surefire generates the XML report and the format does not support tags. As far as the name is concerned, I'd be happy to look into options to improve it for parameterized tests. As this issue is closed, please open a new issue so we can discuss potential options. Thanks!

[bitcoder]

Thanks @marcphilipp .

https://github.com/junit-team/junit5/issues/1519