Publish Test Result: JUnit result with multiple test suites not displayed properly

[ghost]

Hi there,

we're encountering problems using the "Publish test results" tasks under following environment:

• TFS on-premises: 15.117.27024.0
• Private Agent: 2.122.1 running on Windows 7

The test results file (JUnit) contains multiple testsuite elements, which again can contain multiple test cases. The publishing runs without exception. But the "Tests" overview does not group the results by testsuite. Instead it produces a single "parent node", which contains all testcases of all testsuites.

`<?xml version="1.0" encoding="utf-8"?>

Fail1 Fail2 Fail1 Fail2 `  The image shows the single parent named "Unspecified". I would expect two test suites (Testsuite1 and Testsuite2). Greetings

[divais]

Hi,

We don't support Test suites today and thus you see this. Can you please clarify if having test suites as tags meet your need to filter results by Test suites?

[ghost]

Hi,

I've found a workaround for this "problem". If I put a single inside a -tag, it'll work (when grouped by test run). But I have to produce a single file for each testsuite I want to create.

It would help avoiding this workaround (and multiple result files), if I were able to put multiple inside a -tag.

[shanehsu]

Is this going to be supported in the future, or is it on the roadmap?

[kachkaev]

Same problem here. If junit.xml looks like this, testsuite name is ignored.

<testsuites name="jest tests" tests="123" failures="0" time="42">
  <testsuite name="suite 1" errors="0" failures="0" skipped="0" timestamp="2018-12-20T18:58:06" time="6" tests="3">
    <testcase classname="a" name="a" time="1">
    </testcase>
    <testcase classname="b" name="b" time="2">
    </testcase>
    <testcase classname="c" name="c" time="3">
    </testcase>
  </testsuite>
  <testsuite name="suite 2" errors="0" failures="0" skipped="0" timestamp="2018-12-20T18:58:12" time="15" tests="3">
    <testcase classname="x" name="x" time="4">
    </testcase>
    <testcase classname="y" name="y" time="5">
    </testcase>
    <testcase classname="z" name="z" time="6">
    </testcase>
  </testsuite>
</testsuites>

[kachkaev]

Real-world public example: https://www.npmjs.com/package/prettier-plugin-elm

Contents of junit.xml:

<testsuites name="jest tests" tests="18" failures="0" time="18.103">
  <testsuite name="/dist/cache.test.js" errors="0" failures="0" skipped="0" timestamp="2018-12-23T11:16:25" time="14.348" tests="1">
    <testcase classname=" correctly deals with cache" name=" correctly deals with cache" time="13.494">
    </testcase>
  </testsuite>
  <testsuite name="/dist/index.test.js" errors="0" failures="0" skipped="0" timestamp="2018-12-23T11:16:25" time="16.507" tests="17">
    <testcase classname=" formats fixture broken-code.elm" name=" formats fixture broken-code.elm" time="0.902">
    </testcase>
    <testcase classname=" formats fixture broken-code.md" name=" formats fixture broken-code.md" time="0.547">
    </testcase>
    ...
    <testcase classname=" formats fixture only-import.md" name=" formats fixture only-import.md" time="0.417">
    </testcase>
  </testsuite>
</testsuites>

Task:

  - task: PublishTestResults@2
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: $(System.DefaultWorkingDirectory)/junit.xml
    displayName: "Publish test results"
    condition: and(variables.REPORT_TEST_RESULTS_AND_COVERAGE, succeededOrFailed())

Result (test suites are joined, total duration is broken):

HALP 🙏

[divais]

      Is this going to be supported in the future, or is it on the roadmap?

Yes its on the roadmap but not this quarter. It will be of great help if you can request for this feature @ https://developercommunity.visualstudio.com/content/idea/post.html?space=21 to help us prioritize it.

[iamarno]

@divais I still have the same issue. Test suite names are just ignored, only test cases are shown when publishing the test result. Is there any ETA when this will be available?

[lindhe]

But it clearly says in your official documentation that you support test suites (e.g., under Test Results > Outcome):

Failed: if exists /Testsuites/testsuite/testcase/failure

[Additi]

Hi , We have considered this feedback and added this to our backlog. We'll take it up in the coming months. I'll post the updates once its done. Thanks.

[IanKemp]

Hi , We have considered this feedback and added this to our backlog. We'll take it up in the coming months. I'll post the updates once its done. Thanks.

Any update? It's nonsensical that named testcases are supported but named testsuites and/or testsuites are not!

[fdcds]

https://github.com/gotestyourself/gotestsum generates JUnit XML files that look like this:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite tests="3" failures="0" time="1.910000" name="full/package/name">
        <properties>
            <property name="go.version" value="go1.13.7 linux/amd64"></property>
        </properties>
        <testcase classname="full/package/name" name="TestA" time="0.130000"></testcase>
        <testcase classname="full/package/name" name="TestB" time="0.550000"></testcase>
        <testcase classname="full/package/name" name="TestC" time="1.230000"></testcase>
    </testsuite>
    <testsuite tests="3" failures="0" time="1.850000" name="full/package/othername">
        <properties>
            <property name="go.version" value="go1.13.7 linux/amd64"></property>
        </properties>
        <testcase classname="full/package/othername" name="TestA" time="0.130000"></testcase>
        <testcase classname="full/package/othername" name="TestB" time="0.400000"></testcase>
        <testcase classname="full/package/othername" name="TestC" time="1.320000"></testcase>
    </testsuite>
</testsuites>

full/package/name and full/package/othername do not appear in the visual hierarchy displayed by Azure DevOps Pipelines. Instead it uses a flattened representation that shows only the JUnit XML filename and the name of the test (which can be ambiguous, as in my example).

This seems to contradict the documentation, which says that the "Test file" is taken from the /testsuites/testsuite/testcase/Attributes["classname"].Value element in the JUnit XML: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml#result-formats-mapping

[vercellone]

PublishTestResultsV1 outputs properly for me, but not PublishTestResultsV2.

[apis3445]

I want this feature working correctly, the name is not useful

[resried]

Any news, when this is planned?

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[kachkaev]

not stale

[YustinS]

Any chance there is an update for this?

Makes it really hard to use the Test Publisher at scale as it collapses all the test results into a single level, meaning general test case names (such as "Content Header should be X") end up being hard to distinguish which suite they are sourced from, and you have to dive into the JUnit file to actually find out where the issue occurred. This can be worked around by ensuring each name is unique, but it proves to be a hassle in larger test files.

[financelurker]

kicking this issue down the road like a can

Any updates? How long is your roadmap backlog, since we entered the 4th year of this issue.

[IanKemp]

kicking this issue down the road like a can

Any updates? How long is your roadmap backlog, since we entered the 4th year of this issue.

Microsoft isn't interested in building Azure features that are complete. They are only interested in building features enough of the way so that they can claim that Azure supports said feature. So this, like so many other things, will never get fixed.

[anatolybolshakov]

@shailesh-sk could you please take a look?

[svdHero]

Any update on this?

We have just moved from an on-premise Jenkins to Azure DevOps just to discover that in this shiny new cloud world we cannot even review test results anymore due to said flattened list of undistinguishable test cases.

This is a crucial feature!

[Hu3bl]

This is a critical one for us as well! So please give us some updates!

[xiuzhen1103]

This is a critical one for us as well! So please give us some updates +1!

[xiuzhen1103]

Hi,

I've found a workaround for this "problem". If I put a single inside a -tag, it'll work (when grouped by test run). But I have to produce a single file for each testsuite I want to create.

It would help avoiding this workaround (and multiple result files), if I were able to put multiple inside a -tag.

Could you explain where did you put the -tag, please?

[scottsdevelopment]

Very annoying issue.

For now, my workaround includes configuring JUnit to include a more verbose test name.

https://github.com/jest-community/jest-junit#configuration

Environment Variable Name	Reporter Config Name	Description	Default	Possible Injection Values
JEST_JUNIT_TITLE	titleTemplate	Template string for the name attribute of .	"{classname} {title}"	{classname}, {title}, {filepath}, {filename}, {displayName}

I use the following injection: {filepath} / {suitename} / {classname} {title}

"jest-junit": {
    "titleTemplate": "{filepath} / {suitename} / {classname} {title}",
    "outputDirectory": "./coverage",
    "outputName": "junit.xml"
}

Hope this works out for you too, until Azure team finishes these tools.

Regards,

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[silver886]

Any update on this? We are using Azure DevOps Services and all test cases gather in a single test run is not usable when we are reviewing the report.

[MauroMS]

Hey, any updates on this?

[gretel]

four years... duh.

[dnlopes]

Hi , We have considered this feedback and added this to our backlog. We'll take it up in the coming months. I'll post the updates once its done. Thanks.

Yet another person asking for an update on this issue. As said above, this was added to the backlog.

[svdHero]

@dnlopes like you quoted:

We'll take it up in the coming months.

That was 3 years ago. So, in this case, people are just starting to question Microsoft's backlog grooming, I guess. 🤷‍♂️

[joeflan]

Our team would love if this was worked on or we could find a workaround as well. An update would be great

[elearningforce-db]

Any updates? Test results UI looks really inconvenient

[pj-alves]

Can you please add this? It's very important for a lot of teams!

[harleyz]

surprised to find out this isn't supported today.

[gretel]

it's on the backlog since 2019. please be patient 😁

[mega15]

Waiting for this.... too

[farrrb]

Any update here?

[clee704]

Is this repo even being maintained?

[gretel]

Is this repo even being maintained?

it's marketing for peopleless computing 😄

[pj-alves]

Is this repo even being maintained?

Well, releases are coming, but this kind of issues they just don't care... We can just continue to spam 🗡️

[flavian-anselmo]

Hey I also need this

[stefankaufmann]

+1 to see this implemented. Anyone from Microsoft acutally reading this? More than 5 years than it was requested and the previously mentioned link to the roadmap does not work for me.

[flavian-anselmo]

found a work around on how to display junit properly using on ADO ...will share it

[austin-owensby]

I was able to get a somewhat tedious work around for this.

For each test suite I created a separate .xml file and had the PublishTestResults@2 task pick up all of those xml files.

[stefan-muc]

Some promised to share a workaround, but I couldn't find anyone, so I wrote one myself. I'm unsure if I will have to wait until the 6th anniversary of this bug to publish it... (just kidding). Here you are!

It depends on xmlstarlet - a command line XML toolkit

Use your own job.xml - fetching one from the internet as I do here is just an example.

pipeline.yaml:

trigger:
  - '*'

jobs:
- job: fetch_result
  displayName: 'Fetch and Publish Job Result'
  pool: 'Build-Linux'
  workspace:
    clean: outputs

  steps:
  - checkout: self

  - bash: |
      wget -O job.xml https://validation.linaro.org/api/v0.2/jobs/4059960/junit
    workingDirectory: $(Build.BinariesDirectory)
    displayName: Get Job Result from LAVA Instance

  - task: Bash@3
    displayName: 'Split jUnit XML to separate files because of nasty ADO bug'
    inputs:
      filePath: $(Build.SourcesDirectory)/split_junit_testsuites.sh
      arguments: $(Build.BinariesDirectory)/job.xml
      workingDirectory: $(Build.BinariesDirectory)
      noProfile: false

  - task: PublishTestResults@2
    displayName: 'Publish Test Results'
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: '**/test_*.xml'
      searchFolder: $(Build.BinariesDirectory)

split_junit_testsuites.sh:

#!/usr/bin/env bash
#
# This script will split a jUnit XML into files containing only one testsuite.
# This is necessary to work around a many years old bug in Azure DevOps
# See https://github.com/microsoft/azure-pipelines-tasks/issues/7659 for details
#
# License: GPLv2
# Author: stefan-muc (GitHub) in 2024-04

command -v xmlstarlet --version >/dev/null 2>&1 || { echo >&2 "This script requires xmlstarlet in PATH, but couldn't find it. Aborting."; exit 1; }

if [[ $# -ne 1 ]]; then
    echo "Please provide path to jUnit XML as first parameter."
    exit 2
fi

if [[ ! -f "$1" ]]; then
    echo "The file you provided doesn't exist. Aborting."
    exit 3
fi

testsuites=$(xmlstarlet sel -t -v "count(/testsuites/testsuite)" "$1")

if [[ $testsuites -gt 0 ]]; then
    echo "Found $testsuites testsuites"
else
    echo "Couldn't find any testsuite. Exiting."
    exit 3
fi

trap cleanup EXIT
tempdir=$(mktemp -d --tmpdir "ADO-junit-split.XXXXXXXXXX")

if [[ ! -d "$tempdir" ]]; then
    echo "Creating '$tempdir' failed."
    exit 4
fi

tempfile=$tempdir/tempfile.xml
for testsuite in $(seq 1 $testsuites); do
    name=$(xmlstarlet sel -t -m "/testsuites/testsuite[$testsuite]" -v @name "$1")
    echo "Exporting testsuite $testsuite '$name'"

    cp "$1" "$tempfile"
    for i in $(seq $testsuites -1 1); do
        if [[ $i -ne $testsuite ]]; then
            xmlstarlet ed --delete "/testsuites/testsuite[$i]" "$tempfile" > "$tempfile.new"
            mv "$tempfile.new" "$tempfile"
        fi
    done

    mv "$tempfile" "./test_$name.xml"
done

rm "$1" # Delete original file to not export this in a later ADO step

function cleanup()
{
    echo "Cleaning up $tempdir"
    rm -r $tempdir
}

Use at your own risk.

[sgkale]

As a workaround I was able to use PublishTestResults@1 to push multiple test Suites

[eseft]

In case you are here in 2024. If you have a lot xml test results files your testsuites will appear joined on Tests tab in ADO. The reason is the PublishTestResults@2 will join all xml files together if you have more than 100 (see the line in the source code) .

The workaround is following. Under the hood PublishTestResults@2 is using so called Logging Command (at least on Linux agent). I wasn't able to find this command in the azure documentation but seen it during debug run of the pipeline. So one can replace PublishTestResults@2 with simple Bash@3 task:

- task: Bash@3
  inputs:
    targetType: 'inline'
    script: |
      echo "##vso[results.publish type=jUnit;mergeResults=false;publishRunAttachments=false;resultFiles=${FILES_COMMA_SEPARATED};failTaskOnFailedTests=true;failTaskOnFailureToPublishResults=false;testRunSystem=VSTS - PTR;]"

Here FILES_COMMA_SEPARATED comma separated list of full paths to each file. E.g. FILES_COMMA_SEPARATED=/home/vsts/work/results/test_example1.xml,/home/vsts/work/results/test_example2.xml,/home/vsts/work/results/test_example3.xml easy to get with find or smph similar.

I know that task PublishTestResults@2 have the parameter mergeTestResults and it defaults to false but if you have more than 100 files it will be switched to true under the hood.