PublishCodeCoverageResultsV1 cannot handle multiple coverage files

[PureKrome]

Type: Feature

Enter Task Name: PublishCodeCoverageResultsV1

list here (V# not needed):

• https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/PublishCodeCoverageResultsV1

Environment

• Server - Azure Pipelines

• Agent - Hosted

Issue Description

I have a .NET Core project which has multiple tests projects in the solution. Each test project creates a cobertura.xml code coverage file, made by coverlet.

When I run the PublishCodeCoverageResultsV1 task it fails to 'know' that I have multiple files and just picks the first one.

Expected Results: The task is smart enough to merge all files FIRST, then PROCESS the summary file SECOND.

Error/Warning log:

This is the warning in the log file:

##[warning]Multiple file or directory matches were found. Using the first match: /home/vsts/work/1/s/CodeCoverageResults/1fd2623f-e60a-4f9a-b3e0-2d4df53f6440/coverage.cobertura.xml

Sample logs:

Starting: PublishCodeCoverageResults
==============================================================================
Task         : Publish code coverage results
Description  : Publish Cobertura or JaCoCo code coverage results from a build
Version      : 1.160.4
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/publish-code-coverage-results
==============================================================================
##[warning]Multiple file or directory matches were found. Using the first match: /home/vsts/work/1/s/CodeCoverageResults/1fd2623f-e60a-4f9a-b3e0-2d4df53f6440/coverage.cobertura.xml
/usr/bin/dotnet /home/vsts/work/_tasks/PublishCodeCoverageResults_2a7ebc54-c13e-490e-81a5-d7561ab7cd97/1.160.4/netcoreapp2.0/ReportGenerator.dll -reports:**/*.xml -targetdir:/home/vsts/work/_temp/cchtml -reporttypes:HtmlInline_AzurePipelines
2020-08-20T03:47:12: Arguments
2020-08-20T03:47:12:  -reports:**/*.xml
2020-08-20T03:47:12:  -targetdir:/home/vsts/work/_temp/cchtml
2020-08-20T03:47:12:  -reporttypes:HtmlInline_AzurePipelines
2020-08-20T03:47:14: Writing report file '/home/vsts/work/_temp/cchtml/index.html'
2020-08-20T03:47:14: Report generation took 1.9 seconds
Generated code coverage html report: /home/vsts/work/_temp/cchtml
Reading code coverage summary from '/home/vsts/work/1/s/CodeCoverageResults/1fd2623f-e60a-4f9a-b3e0-2d4df53f6440/coverage.cobertura.xml'
Async Command Start: Publish code coverage
Publishing coverage summary data to TFS server.
 Lines- 352 of 3960 covered.
 Branches- 69 of 1337 covered.
Modifying Cobertura Index file
Publishing code coverage files to TFS server.
Uploading 173 files
Total file: 173 ---- Processed file: 96 (55%)
File upload succeed.
Published '/home/vsts/work/_temp/cchtml' as artifact 'Code Coverage Report_3911'
Async Command End: Publish code coverage
Finishing: PublishCodeCoverageResults

Thank you kindly!

[phanikmmsft]

@nishantms please look into this

[drdamour]

@PureKrome FWIW this is a known limitation of v1 per the docs https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results?view=azure-devops#is-code-coverage-data-merged-when-multiple-files-are-provided-as-input-to-the-task-or-multiple-tasks-are-used-in-the-pipeline

the v2 seems to handle this https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/PublishCodeCoverageResultsV2/task.json but it's not released on devops #13257

note you can merge them yourself with something like this

    # https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/powershell
    # report generator docs https://github.com/danielpalme/ReportGenerator#usage--command-line-parameters
    # The publish task can only handle 1 coverage file for the summary value shown in test & coverage area of a build summary
    # NOTE: the publish task DOES handle multiple for the actual report, but this combining makes it simpler
    - task: PowerShell@2
      displayName: combine coverage
      inputs:
        targetType: 'inline'
        script: |
          Set-PSDebug -trace 1

          dotnet tool install -g dotnet-reportgenerator-globaltool

          reportgenerator "-reports:$(Agent.TempDirectory)/**/coverage.cobertura.xml" "-targetdir:$(Agent.TempDirectory)" -reporttypes:Cobertura

    # https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results
    - task: PublishCodeCoverageResults@1
      displayName: 'publish coverage'
      inputs:
        codeCoverageTool: 'Cobertura'
        # this is the combined file output from combine coverage task, as of 2021-Q1 you can't specify the file name, it's just named to the type
        summaryFileLocation: '$(Agent.TempDirectory)/Cobertura.xml'

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[IvanAlekseev]

Hey little bot, why would you close an issue when it is not fixed?

[svdHero]

Are there any plans to implement the coverage file merging any time soon? We really need this. This is a crucial feature.

[Sharparam]

Confusingly, in the documentation (which references v1 of the task), the text under "Arguments" for summaryFileLocation claims that merging is supported.

(Required) Path of the summary file containing code coverage statistics, such as line, method, and class coverage. Multiple summary files will be merged into a single report. The value may contain minimatch patterns. For example: $(System.DefaultWorkingDirectory)/MyApp/**/site/cobertura/coverage.xml

(emphasis mine)

Only further down the page in the FAQ does it say it's not supported, leading to conflicting statements.

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[IvanAlekseev]

not stale

[dxynnez]

Confusingly, in the documentation (which references v1 of the task), the text under "Arguments" for summaryFileLocation claims that merging is supported.

(Required) Path of the summary file containing code coverage statistics, such as line, method, and class coverage. Multiple summary files will be merged into a single report. The value may contain minimatch patterns. For example: $(System.DefaultWorkingDirectory)/MyApp/**/site/cobertura/coverage.xml

(emphasis mine)

Only further down the page in the FAQ does it say it's not supported, leading to conflicting statements.

I got confused by the same thing... I decided to still give it a try and it did seem to generate correct aggregated coverage report. The only downside I am seeing now is the annoying warning ([warning]Multiple file or directory matches were found. Using the first match)... MSFT never hesitates to surprise me, this time it seems to be in a good way, though.

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[drdamour]

still not stale

[cospeedster]

Hey, today i stumbled over the same problem.

I've found a workaround using ReportGenerator (GitHub). Maybe it's useful for someone who cames here.

- bash: |
    dotnet test \
      --logger:"nunit;LogFilePath=$(Agent.TempDirectory)/TestResults/NUnit-TestResults.xml" \
      --collect:"XPlat Code Coverage;Format=Cobertura,opencover" \
      --results-directory $(Agent.TempDirectory)/TestResults \
      --nologo \
      --verbosity normal
  displayName: dotnet test

- bash: echo "##vso[task.setvariable variable=disable.coverage.autogenerate;]true"
  displayName: Disable autogeneration of code coverage report

- task: reportgenerator@5
  inputs:
    reports: '$(Agent.TempDirectory)/TestResults/**/coverage.cobertura.xml'
    targetdir: '$(Agent.TempDirectory)/coveragereport'
    reporttypes: 'HtmlInline_AzurePipelines_Dark;Cobertura'
  displayName: 'Generate code coverage report'

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage report'
  condition: succeededOrFailed()
  inputs:
    codeCoverageTool: 'Cobertura'
    summaryFileLocation: '$(Agent.TempDirectory)/coveragereport/Cobertura.xml'
    reportDirectory: '$(Agent.TempDirectory)/coveragereport'
    failIfCoverageEmpty: true

[abatishchev]

Is it the same issue as https://github.com/microsoft/azure-pipelines-tasks/issues/10353?

Also have you tried PublishCodeCoverageResults@2 yet? Does it improve anything in these regards?

[cospeedster]

I have no chance to try because PublishCodeCoverageResults@2 is not available on Azure DevOps Server.

[kkazala]

It works, but report is different #10353

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[abatishchev]

Since the v2 of this task was released earlier last year, I think this issue is now moot.

[iansalazar444]

A bit warning to future readers - as of now, V2 does not support / generate branch code coverage. I'm sticking to V1 for now with the work around stated above