search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.47k stars 2.6k forks source link

Merge coverage reports from multiple runs #10353

Open pkuczynski opened 5 years ago

pkuczynski commented 5 years ago

Required Information

Type: bug Enter Task Name: PublishCodeCoverageResults@1

Environment

Issue Description

Task PublishCodeCoverageResults@1 supports reports from multiple files, but the summary just shows the first match.

Task logs

The behavior for a task configured with a wildcard path for the coverage file is a little inconsistent:

      - task: PublishCodeCoverageResults@1
        inputs:
          codeCoverageTool: cobertura
          summaryFileLocation: $(System.DefaultWorkingDirectory)/packages/*/build/coverage/coverage.cobertura.xml

In this example, there are few coverage.cobertura.xml files from separate test runs. Unfortunately, I receive warning:


##[warning]Multiple file or directory matches were found. Using the first match: ```

Ideally, the summary would show the coverage for each assembly separately. Because I'm working in a monorepo and would like to trace multiple folders separately.

A similar issue has been reported in https://github.com/MicrosoftDocs/vsts-docs/issues/4135
mhamilton723 commented 5 years ago

+1 on this issue!

pderaaij commented 5 years ago

This would be a great addition when working with a monorepo.

MisinformedDNA commented 5 years ago

Is there a workaround for this?

pkuczynski commented 5 years ago

Not that I know of :(

juliocr-ciandt commented 5 years ago

+1

boyanio commented 4 years ago

I use a custom script to merge all reports into a single one. In this gist you can see an example using istanbul

avilde commented 4 years ago

I use jest-junit and cobertura reports both from front-end (create react app, jest) and back-end (.NET Core 3.0) but there is no merge option for both reports. What a pity.

+1

jhonnymertz commented 4 years ago

Is there any update on this issue? I'm working on a monorepo and the fact that it cannot collect all the coverage reports limits the usage of this information in dashboards, for example. All the existing widgets to report code coverage in ADO dashboards rely on this data...

MisinformedDNA commented 4 years ago

To get around this, we can merge all the files using ReportGenerator. This is what worked for me:

  1. (Optional) If the code coverage is coming from different jobs, publish the coverage files as artifacts and then have a later job download all the artifacts

  2. Create a report from all coverage files and publish

    
- task: reportgenerator@4
inputs:
reports: **/coverage.cobertura.xml;artifact/**/cobertura-coverage.xml
targetdir: $(Pipeline.Workspace)/coveragereport

This has worked well for me.

andreujuanc commented 4 years ago

Work around is good and all, but in some orgs, it can take quite a while until you get a marketplace extension approved, if at all. 100% should be handled by Azure Pipelines and not some random extension.

drdamour commented 3 years ago

fwiw it looks like this might have been fixed with the v2 of this task at https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/PublishCodeCoverageResultsV2/task.json with https://github.com/microsoft/azure-pipelines-tasks/pull/11144 but it hasn't been published to devops...not sure why was also asked in #13257

drdamour commented 3 years ago

@MisinformedDNA where's that reportgenerator task come from?

MisinformedDNA commented 3 years ago

@drdamour https://marketplace.visualstudio.com/items?itemName=Palmmedia.reportgenerator

ByteDev commented 3 years ago

Using reportgenerator in the way described above doesn't fully work for me. I get the following warning from the PublishCodeCoverageResults@1 task: [warning]No code coverage results were found to publish..

However, if i set summaryFileLocation: '$(Build.SourcesDirectory)/**/coverage.cobertura.xml' then everything works but I'm back to getting the misleading warning: Multiple file or directory matches were found. Using the first match....

MisinformedDNA commented 3 years ago

My guess is that something either is in the wrong place or is looking in the wrong place.

Can you share your YAML?

msrinivascharan commented 3 years ago

I have a project with three different modules, two Java and one Python. all three modules generate Cobertura based code coverage reports. I am able to merge all three into one using @danielpalme "ReportGenerator" extension. able to apply gating condition using merged coverage report. Please refer this.

mike-bagheri commented 3 years ago

The reportgenerator can merge the results, and then I had to specify the reporttypes and include Cobertura to make it work

- task: DotNetCoreCLI@2
  displayName: 'Create code coverage report'
    inputs:
      command: custom
      custom: tool
      arguments: 'run reportgenerator -reports:$(Build.SourcesDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)\CodeCoverage -reporttypes:HtmlInline_AzurePipelines;Cobertura'

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage report'
    inputs:
      codeCoverageTool: 'cobertura'
       summaryFileLocation: '$(Build.SourcesDirectory)\CodeCoverage\*cobertura.xml'
jack4it commented 3 years ago

The reportgenerator can merge the results, and then I had to specify the reporttypes and include Cobertura to make it work

- task: DotNetCoreCLI@2
  displayName: 'Create code coverage report'
    inputs:
      command: custom
      custom: tool
      arguments: 'run reportgenerator -reports:$(Build.SourcesDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)\CodeCoverage -reporttypes:HtmlInline_AzurePipelines;Cobertura'

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage report'
    inputs:
      codeCoverageTool: 'cobertura'
       summaryFileLocation: '$(Build.SourcesDirectory)\CodeCoverage\*cobertura.xml'

In my case, I had to add the reportDirectory param as well

          - task: PublishCodeCoverageResults@1
            displayName: Publish code coverage results
            inputs:
              codeCoverageTool: Cobertura
              summaryFileLocation: "$(System.DefaultWorkingDirectory)/testresults/coverage/reports/Cobertura.xml"
              reportDirectory: "$(System.DefaultWorkingDirectory)/testresults/coverage/reports/"

also add a job level variable to disable the publisher generation of coverage reports. so that the self generated reports will be uploaded instead

      - job: Docker
        displayName: Capture version, build/push docker image
        continueOnError: false
        variables:
          disable.coverage.autogenerate: true
        steps:
svdHero commented 2 years ago

Hi there,

when I try to use the workaround above as in:

    pool:
      vmImage: ubuntu-20.04

    variables:
      disable.coverage.autogenerate: 'true' # for report generator

    steps:
    # more tasks here...

    - task: DotNetCoreCLI@2
      displayName: 'Install ReportGenerator'
      inputs:
        command: custom
        custom: tool
        arguments: 'install --global dotnet-reportgenerator-globaltool --version 4.8.12'

    - task: DotNetCoreCLI@2
      displayName: 'Merge code coverage files'
      inputs:
        command: custom
        custom: tool
        arguments: 'run reportgenerator -reports:$(Build.SourcesDirectory)/build_reports/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/build_reports/code_coverage -reporttypes:HtmlInline_AzurePipelines;Cobertura'

I get the error message

Cannot find a tool in the manifest file that has a command named 'reportgenerator'.

Any ideas what's wrong here? @mike-bagheri and @jack4it , how did you guys install reportgenerator? I'd appreciate any advice. Thank you so much in advance.

edit: Corrected false to true in disable.coverage.autogenerate which, however, is unrelated to my question.

IvanAlekseev commented 2 years ago

We just use a script for that

- script: dotnet tool install --global dotnet-reportgenerator-globaltool
  displayName: Install ReportGenerator

- script: reportgenerator -reports:$(Agent.TempDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/CodeCoverage -reporttypes:Cobertura
  displayName: Create Code coverage report
mike-bagheri commented 2 years ago

Hi there,

when I try to use the workaround above as in:

    pool:
      vmImage: ubuntu-20.04

    variables:
      disable.coverage.autogenerate: 'false' # for report generator

    steps:
    # more tasks here...

    - task: DotNetCoreCLI@2
      displayName: 'Install ReportGenerator'
      inputs:
        command: custom
        custom: tool
        arguments: 'install --global dotnet-reportgenerator-globaltool --version 4.8.12'

    - task: DotNetCoreCLI@2
      displayName: 'Merge code coverage files'
      inputs:
        command: custom
        custom: tool
        arguments: 'run reportgenerator -reports:$(Build.SourcesDirectory)/build_reports/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/build_reports/code_coverage -reporttypes:HtmlInline_AzurePipelines;Cobertura'

I get the error message

Cannot find a tool in the manifest file that has a command named 'reportgenerator'.

Any ideas what's wrong here? @mike-bagheri and @jack4it , how did you guys install reportgenerator? I'd appreciate any advice. Thank you so much in advance.

From the extension in Azure DevOps, installed this

image

You may also need to add this

- task: DotNetCoreCLI@2
  displayName: 'Create .NET local manifest'
  inputs:
    command: custom
    custom: new
    arguments: 'tool-manifest
danielpalme commented 2 years ago

There a two options:

svdHero commented 2 years ago

Thank you @danielpalme , but when I do a

    - task: DotNetCoreCLI@2
      displayName: 'Install ReportGenerator'
      inputs:
        command: custom
        custom: new
        arguments: 'tool-manifest'

    - task: DotNetCoreCLI@2
      displayName: 'Install ReportGenerator'
      inputs:
        command: custom
        custom: tool
        arguments: 'install --global dotnet-reportgenerator-globaltool --version 4.8.12'

    - task: DotNetCoreCLI@2
      displayName: 'Merge code coverage files'
      inputs:
        command: custom
        custom: tool
        arguments: 'run reportgenerator -reports:$(Build.SourcesDirectory)/build_reports/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/build_reports/code_coverage -reporttypes:HtmlInline_AzurePipelines;Cobertura'

I get a different pipeline error saying

Cannot find a tool in the manifest file that has a command named 'reportgenerator'.

Any idea what's wrong here? Do I always have to call reportgenerator instead of dotnet run reportgenerator?

edit: Corrected code snippet above.

I will also go and try your AzureDevops Extension.

danielpalme commented 2 years ago

@svdHero I just tried it myself, the following works for me:

- task: DotNetCoreCLI@2
  displayName: 'Install ReportGenerator'
  inputs:
    command: custom
    custom: tool
    arguments: 'install --global dotnet-reportgenerator-globaltool --version 4.8.12'

- script: reportgenerator "-reports:$(Build.SourcesDirectory)/build_reports/**/coverage.cobertura.xml"" -targetdir:$(Build.SourcesDirectory)/build_reports/code_coverage" -reporttypes:HtmlInline_AzurePipelines;Cobertura
  displayName: Merge code coverage files
github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

drdamour commented 2 years ago

Bamp/not stale

james1301 commented 2 years ago

@acesiddhu is there an update on this?

purijatin commented 2 years ago

This is a pain point. We have a repo that has java, python and dotnet modules. And cumbersome to merge all of them to a single report.

inloox-konrad commented 2 years ago

We've now run into this as well. The doc at https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results?view=azure-devops clearly states that "Multiple summary files will be merged into a single report.", but that is patently untrue. At the least the doc should be fixed to reflect the reality thay this feature is not yet included. Finding this out the hard way is time-consuming.

jerone commented 2 years ago

@drdamour commented on Jan 13, 2021, 8:12 PM GMT+1:

fwiw it looks like this might have been fixed with the v2 of this task at https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/PublishCodeCoverageResultsV2/task.json with #11144 but it hasn't been published to devops...not sure why was also asked in #13257

1.5 year later and v2 is still not available. :(

aolszowka commented 2 years ago

Keeping the issue alive; as mentioned by others still no fix, and based on this comment https://github.com/microsoft/azure-pipelines-tasks/issues/15160#issuecomment-902632589 doesn't seem like there is a plan to merge in the v2 version of this task which does have a combiner?

The suggested workaround of using ReportGenerator (https://github.com/danielpalme/ReportGenerator) does work but is a bit of a hack in the sense that the last user of PublishCodeCoverageResults is the winner. This can be difficult if you have a complicated pipeline that leverages templates and requires a significant amount of coordination.

abatishchev commented 1 year ago

Same here. Only 1 out of many tests projects is included into the results.

I tried adding an additional task mentioned on Stack Overflow:

- task: reportgenerator@4
  displayName: 'Merge code coverage reports'
  inputs:
    reports: '$(Agent.TempDirectory)\TestResults\**\coverage.cobertura.xml'
    targetdir: '$(Agent.TempDirectory)\TestResults\Merged\'
    reporttypes: Cobertura
    verbosity: Verbose

but reportgenerator is not and can't be installed to my ADO project.

danielpalme commented 1 year ago

but reportgenerator is not and can't be installed to my ADO project.

You have to hit the "Get it free" button on this website: https://marketplace.visualstudio.com/items?itemName=Palmmedia.reportgenerator

Then the task should be available (you should use reportgenerator@5)

abatishchev commented 1 year ago

You have to hit the "Get it free" button on this website:

Unfortunately I can't, my company's policy prohibits installing third-party extensions. Technically there is a process but it's near-impossible to get through.

I'm trying to work it around by install it as a global tool and run directly. But so far it doesn't make any difference: all tasks succeed, build succeeds, code coverage report stays the same.

I have multiple test projects which I run simultaneously by passing *.Tests.csproj. Is the resulting code coverage results file's name unique?

- task: DotNetCoreCLI@2
  displayName: 'Run tests'
  inputs:
    command: test
    projects: '**\*Tests.csproj'
    arguments: '--no-build --no-restore --configuration ${{ parameters.configuration }} --settings $(RepoRoot)\.runsettings --blame --collect "Code coverage" --logger "trx" --results-directory $(Agent.TempDirectory)\TestResults\ /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Agent.TempDirectory)\TestResults\'
    publishTestResults: false

- task: DotNetCoreCLI@2
  displayName: 'Install reportgenerator'
  inputs:
    command: custom
    custom: tool
    arguments: 'install --global dotnet-reportgenerator-globaltool'

- task: PowerShell@2
  displayName: 'Merge code coverage reports'
  inputs:
    targetType: inline
    script: |
      $source = '$(Agent.TempDirectory)\TestResults\**\coverage.cobertura.xml'
      $target = '$(Agent.TempDirectory)\TestResults\Merged\'
      & reportgenerator.exe -reports:$source "-targetdir:$target" "-reporttypes:HtmlInline_AzurePipelines;Cobertura"

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage'
  condition: always()
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(Agent.TempDirectory)\TestResults\Merged\**\*.xml'

Where's the mistake?

danielpalme commented 1 year ago

This should work 

disable.coverage.autogenerate: 'true'

...

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage'
  condition: always()
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(Agent.TempDirectory)\TestResults\Merged\Cobertura.xml'
    reportDirectory: '$(Agent.TempDirectory)\TestResults\Merged'

The environment variable "disable.coverage.autogenerate" is important, otherwise PublishCodeCoverageResults will regenerate the coverage HTML report.

abatishchev commented 1 year ago

hi @danielpalme, thank you for your help! unfortunately I'm still unable to get it working.

What worked: the format has changed and it looks much nicer:

image

However it still shows only the first test project's results.

My YAML:

- task: DotNetCoreCLI@2
  displayName: 'Run tests'
  inputs:
    command: test
    projects: '**\*Tests.csproj'
    arguments: '--no-build --no-restore --configuration ${{ parameters.configuration }} --settings $(RepoRoot)\.runsettings --blame --collect "Code coverage" --logger "trx" --results-directory $(Agent.TempDirectory)\TestResults\ /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Agent.TempDirectory)\TestResults\'
    publishTestResults: false

- task: DotNetCoreCLI@2
  displayName: 'Install reportgenerator'
  inputs:
    command: custom
    custom: tool
    arguments: 'install --global dotnet-reportgenerator-globaltool'

- task: PowerShell@2
  displayName: 'Merge code coverage reports'
  inputs:
    targetType: inline
    script: |
      Write-Host '##vso[task.setVariable variable=disable.coverage.autogenerate;isSecret=false]true'

      $source = '$(Agent.TempDirectory)\TestResults\**\coverage.cobertura.xml'
      $target = '$(Agent.TempDirectory)\TestResults\Merged\'

      & reportgenerator.exe -reports:$source -targetdir:$target -reporttypes:"HtmlInline_AzurePipelines;Cobertura"

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage'
  condition: always()
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(Agent.TempDirectory)\TestResults\Merged\**\*.xml'
    reportDirectory: '$(Agent.TempDirectory)\TestResults\Merged'

Can you please take a look and see if I've missed anything?

aolszowka commented 1 year ago

@abatishchev You beat me to the punch I was just going to reply to say that you can install the tool manually via Dotnet-Tool and merge it, you've got it and your solution looks more or less like what we use internally. However as I mentioned above:

The suggested workaround of using ReportGenerator (https://github.com/danielpalme/ReportGenerator) does work but is a bit of a hack in the sense that the last user of PublishCodeCoverageResults is the winner. This can be difficult if you have a complicated pipeline that leverages templates and requires a significant amount of coordination.

Be careful!

danielpalme commented 1 year ago

Can you please take a look and see if I've missed anything?

I hope. Could you please add the following argument to reportgenerator.exe: -verbosity:verbose Then you could check the output of that task.

For every coverage file that gets processed the following line should appear in the messages: Loading report '%%PATH%%' x/y in memory

Do several coverage files get processed?

abatishchev commented 1 year ago

@danielpalme

Not good :(

2023-01-17T20:53:46: Loading report 'C:\__w\_temp\TestResults\coverage.cobertura.xml' 1/1 in memory

Here's the "Run tests" task logs:

Attachments:
  C:\__w\_temp\TestResults\d656b065-66c2-4f1d-a967-7ec1cda8404a\ContainerAdministrator_C13A536809CD_2023-01-17.20_45_39.coverage
Passed!  - Failed:     0, Passed:    82, Skipped:     0, Total:    82, Duration: 27 s - Common.Tools.Tests.dll (net6.0)

Calculating coverage result...
  Generating report 'C:\__w\_temp\TestResults\coverage.cobertura.xml'

Apparently, it's the "last run wins": the same file is overwritten by each consequent test project.

danielpalme commented 1 year ago

In most of my projects I use the package coverlet.collector

Then I run the tests with the following command (only important arguments added here): dotnet test --collect:"XPlat Code Coverage"

This results in cobertura file in each test project in a subfolder named TestResults\%%GUID%%\coverage.cobertura.xml

Then I pass all cobertura files to ReportGenerator with the following argument:

-reports:*Tests\TestResults\*\coverage.cobertura.xml

abatishchev commented 1 year ago

I've changed the "run tests" to this:

- task: DotNetCoreCLI@2
  displayName: 'Run tests'
  inputs:
    command: test
    projects: '**\*Tests.csproj'
    arguments: '--no-build --no-restore --configuration ${{ parameters.configuration }} --settings $(RepoRoot)\.runsettings --blame --collect "XPlat Code Coverage" --logger "trx" --results-directory $(Agent.TempDirectory)\TestResults\ /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Agent.TempDirectory)\TestResults\'
    publishTestResults: false

i.e. only from --collect "Code coverage" to --collect "XPlat Code Coverage" and it made no difference:

Results File: C:\__w\_temp\TestResults\ContainerAdministrator_AED0B744811A_2023-01-17_23_46_43.trx

Passed!  - Failed:     0, Passed:     2, Skipped:     0, Total:     2, Duration: 81 ms - ConfigurationTests.dll (net6.0)

Calculating coverage result...
  Generating report 'C:\__w\_temp\TestResults\coverage.net6.0.cobertura.xml'
danielpalme commented 1 year ago

Are you sure that the "Run tests" task executes all test projects? I'm not familiar with all the inputs of that task and your supplied arguments.

abatishchev commented 1 year ago

Yes, the Tests tab shows all 551 have been executed every time. The problem is in the code coverage file being overwritten every time.

How come yours gets a guid appended to its path?

danielpalme commented 1 year ago

Maybe you could try to remove all these arguments. I guess they are affecting the target directory. I'm using the setup described here: https://reportgenerator.io/getstarted#net

Instead of arguments: '--no-build --no-restore --configuration ${{ parameters.configuration }} --settings $(RepoRoot)\.runsettings --blame --collect "XPlat Code Coverage" --logger "trx" --results-directory $(Agent.TempDirectory)\TestResults\ /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Agent.TempDirectory)\TestResults\'

You could start with: arguments: '--no-build --no-restore --configuration ${{ parameters.configuration }} --settings $(RepoRoot)\.runsettings --blame --collect "XPlat Code Coverage"'

abatishchev commented 1 year ago

hi @danielpalme, sorry - just coming back to this issue.

What doesn't work for me: changing it to "XPlat Code Coverage":

Data collection : Unable to find a datacollector with friendly name 'XPlat Code Coverage'.
Data collection : Could not find data collector 'XPlat Code Coverage'

However what seems to make a difference:

from

<DebugType>pdbonly</DebugType>

to

<DebugType>full</DebugType>
SymbioticKilla commented 1 year ago

Hi,

is there any ETA or priority for this issue? We use 3rdParty generator as workaround, but it would be nice to have stable out-of-the-box solution from Microsoft. Thanks!

kkazala commented 1 year ago

PublishCodeCoverageResults@2 works for me without any additional report merging The report is waay simpler though.

The old one: image

vs the new one: image

frittsy commented 1 year ago

Finally, all my years of tracking this issue have paid off. v2 is showing up in Azure DevOps for me as well and is listed in their documentation as a task, as of 7/10/2023.

felipepessoto commented 1 year ago

It only merge files before publishing? If I have multiple jobs it means it won't work? For example, PublishTestResults works fine with multiple jobs if you set mergeTestResults=true

github-actions[bot] commented 7 months ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days