search

danielpalme / ReportGenerator

ReportGenerator converts coverage reports generated by coverlet, OpenCover, dotCover, Visual Studio, NCover, Cobertura, JaCoCo, Clover, gcov or lcov into human readable reports in various formats.
https://reportgenerator.io
Apache License 2.0
2.56k stars 279 forks source link

[Question] Up-to-date overview: How to use ReportGenerator with Azure DevOps? #646

Closed georg-jung closed 6 months ago

georg-jung commented 7 months ago

Thanks for creating ReportGenerator! I think it provides a great overview of code coverage and I really like all the integration options.

I got however a bit confused about all the options to use it and the best way to use ReportGenerator with Azure DevOps as of now, 2024. I've seen many issues discussing this topic including #398, but many of them are outdated as of 2024. Maybe it could be helpful for new users to have one central up-to-date page with guidance (which the other places including the Azure DevOps Marketplace page could link to). I'd be happy to contribute in that direction if that makes sense, but I guess at this point the best thing I can do is asking the questions I thought about when getting started with Code Coverage in Azure DevOps.

My Goals (decreasing prio)

Options I came across

What I think I understood

Questions

- task: DotNetCoreCLI@2
  inputs:
    command: test
    projects: '$(solution)'
    arguments: '--no-restore --configuration Release -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
    testRunTitle: 'release'
    publishTestResults: true
  displayName: dotnet test Release
variables:
  solution: './Xyz.sln'
  disable.coverage.autogenerate: 'true' # https://github.com/microsoft/azure-pipelines-tasks/issues/4945#issuecomment-823832527

jobs:
  - job: Build
    pool:
      vmImage: ubuntu-22.04
    steps:
    - script: dotnet restore "$(solution)" /p:ContinuousIntegrationBuild=true
      displayName: dotnet restore

    - script: dotnet build --no-restore -c Release "$(solution)"  /p:ContinuousIntegrationBuild=true
      displayName: dotnet build

    - task: DotNetCoreCLI@2
      inputs:
        command: test
        projects: '$(solution)'
        arguments: '--no-restore --configuration Debug -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
        testRunTitle: 'debug'
        publishTestResults: true
      displayName: dotnet test Debug

    - task: DotNetCoreCLI@2
      inputs:
        command: test
        projects: '$(solution)'
        arguments: '--no-restore --configuration Release -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
        testRunTitle: 'release'
        publishTestResults: true
      displayName: dotnet test Release

    - task: PublishCodeCoverageResults@2
      displayName: 'Publish code coverage report'
      inputs:
        summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
        failIfCoverageEmpty: true

I hope I didn't miss any obvious places to research this myself. Thanks again for creating ReportGenerator!

danielpalme commented 7 months ago

Thanks for your detailed issue and the research behind it.

You are right, there are several options. My experience is, that PublishCodeCoverageResults@1 or PublishCodeCoverageResults@2 are required to get the "Code Coverage" tab

PublishCodeCoverageResults@1 uses an (outdated) version of ReportGenerator internally. If you don't specify disable.coverage.autogenerate: 'true' it will (re-)generate the coverage report based on the Coberatura file. If you want to use the latest version of ReportGenerator you should always set disable.coverage.autogenerate: 'true'.

PublishCodeCoverageResults@2 does not use ReportGenerator internally. If you have different information, I would be interested.
Instead it generates it's own report which looks like this: image As far as I know it's not possible to publish a full report with PublishCodeCoverageResults@2, since there is no Report Directory parameter.

Regarding your questions:

  • Whats the minimal, optimal but full example to use ReportGenerator with Azure DevOps today? I've seen https://reportgenerator.io/usage but in my testing using only the reportgenerator task didn't produce e.g. the Code Coverage tab.

Correct. You'll have to combine it with PublishCodeCoverageResults@1. See also: https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-extension

You could generate two reports in separate directories. Then you would probably have to rename the summary.html file to index.html before publishing the it with PublishCodeCoverageResults@1

- task: reportgenerator@5
  displayName: ReportGenerator
  inputs:
    reports: 'src\target\reports\coverage\coverage.xml'
    targetdir: '$(Build.SourcesDirectory)/coveragereport'
    reporttypes: 'HtmlInline_AzurePipelines;HtmlSummary;Cobertura'
    customSettings: 'settings:createSubdirectoryForAllReportTypes=true'

# Now rename file `$(Build.SourcesDirectory)/coveragereport/HtmlSummary/summary.html` to `$(Build.SourcesDirectory)/coveragereport/HtmlSummary/index.html`

# Create artifact of directory $(Build.SourcesDirectory)/coveragereport/HtmlInline_AzurePipelines

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage results'
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(Build.SourcesDirectory)/coveragereport/Cobertura/Cobertura.xml'
    reportDirectory: '$(Build.SourcesDirectory)/coveragereport/HtmlSummary'
  env:
    DISABLE_COVERAGE_AUTOGENERATE: 'true'
  • disable.coverage.autogenerate: 'true' doesn't seem to work for me with PublishCodeCoverageResults@2. Is there a workaround for V2 too? Is it better to use V1 with reportgenerator@5?

As mentioned above, I'm pretty sure that you have to use PublishCodeCoverageResults@1. disable.coverage.autogenerate: 'true' does not work here. And there is no option to specify a report directory any more. Am I missing something?!

  • I have a DotNetCoreCLI@2 test task like the following. The PublishCodeCoverageResults@2 seems to first publish ReportGenerator's report, which is later overwritten. How to fix this while still publishing the trx test results?

I guess that's not possible. See also your next question.

  • My current production pipeline config is similar to the following. It does fulfil most of my requirements except that the contents of "Code Coverage" are replaced by an autogenerated report when the pipleine finishes. How to get that right?

This is also my experience. See my answers above. I always use PublishCodeCoverageResults@1 instead of PublishCodeCoverageResults@2.

georg-jung commented 7 months ago

Thanks for your detailed response!

PublishCodeCoverageResults@2 does not use ReportGenerator internally. If you have different information, I would be interested.

My example pipeline above with 

    - task: DotNetCoreCLI@2
      inputs:
        command: test
        projects: '$(solution)'
        arguments: '--no-restore --configuration Release -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
        testRunTitle: 'release'
        publishTestResults: true
      displayName: dotnet test Release

    - task: PublishCodeCoverageResults@2
      displayName: 'Publish code coverage report'
      inputs:
        summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
        failIfCoverageEmpty: true

- and notably without any reportgenerator@5 task - leads to the following behaviour:

=> Thus, a V2 disable.coverage.autogenerate: 'true' might be quite interesting.

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/dotnet-core-cli-v2?view=azure-pipelines reads

publishTestResults: true # boolean. Optional. Use when command = test. Publish test results and code coverage. Default: true.

Not sure how this might affect the behaviour.

I changed my pipeline now like this and it works as exected. The Code Coverage tab does contain the ReportGenerator report even after the end of the job (and also still accounts for .netconfig).

    - task: DotNetCoreCLI@2
      inputs:
        command: test
        projects: '$(solution)'
        arguments: '--no-restore --configuration Release -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
        testRunTitle: 'release'
        publishTestResults: true
      displayName: dotnet test Release

    # https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-extension
    - task: reportgenerator@5
      displayName: '[Coverage] ReportGenerator'
      inputs:
        reports: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
        targetdir: '$(Agent.TempDirectory)/coveragereport'
        reporttypes: 'HtmlInline_AzurePipelines;Cobertura;Badges'
        assemblyfilters: '-xunit*'

    - task: PublishCodeCoverageResults@1
      displayName: '[Coverage] Publish results'
      inputs:
        codeCoverageTool: Cobertura
        summaryFileLocation: '$(Agent.TempDirectory)/coveragereport/Cobertura.xml'
        reportDirectory: '$(Agent.TempDirectory)/coveragereport'
      env:
        DISABLE_COVERAGE_AUTOGENERATE: 'true'

Thanks for pointing me to https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-extension. I actually overlooked this first (even though it's linked in the README 🙄 - I guess there are just many places providing pieces of information, some of which are more up to date while others aren't).

danielpalme commented 7 months ago

=> Thus, a V2 disable.coverage.autogenerate: 'true' might be quite interesting.

There is no such option. I looked at the source code: https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/PublishCodeCoverageResultsV2

Internally PublishCodeCoverageResults@1 uses https://github.com/microsoft/azure-pipelines-coveragepublisher to create the coverage reports.

rikrak commented 7 months ago

FWIW I would have benefitted from such a guide. It took me some time to understand that the PublishCodeCoverageResults@2 task was flawed, and the older version is preferred in my scenario (which was similar to the one described above). The main difficulty I faced in researching the production of a Devops Pipeline code coverage report, was that I didn't realise that a lot of the examples out in the wild (using the V2 task) are quite simplistic and don't cover what I'd consider a "real-world" scenario. In addition I thought any examples using the older V1 task were out-of-date :-/

danielpalme commented 7 months ago

@rikrak Yes there are several options.

But the overall process is documented here: https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-vsts

The FAQs also answer the most common question: https://github.com/danielpalme/ReportGenerator/wiki/FAQ#azure-devops-extension-seems-to-ignore-my-settings

jchenathub24 commented 6 months ago
  • As soon as the PublishCodeCoverageResults@2 task finished (but not the whole job/pipeline), the Code Coverage tab contains a ReportGenerator report. When specifying a license key using .netconfig it even contains a pro version report.
  • At the same time, a "Code Coverage Report_" artifact is created.
  • As soon as the pipeline(/job?) finishes, the Code Coverage tab's content is replaced with the report format that can be seen in your screenshot.

@georg-jung I come cross the same behaviour.

When I test steps like:

  1. dotnet test: generate coverage file
  2. using PublishCodeCoverageResults@2

I can see that PublishCodeCoverageResults@2 internally use ReportGenerator, it shows a footer in the report saying that. But later, the whole report is override when pipeline finished. Feel like a bug for that task.

Also, I try the disable.coverage.autogenerate: 'true', it does not change any outcome from PublishCodeCoverageResults@2.

So the options I have tested and working for me: Option1:

  1. dotnet test: generate coverage file.
  2. reportgenerator: generate report file from coverage file in step 1. (ReportGenerator 5.2.1.0 by the time of this reply)
  3. disable re-generate report: by setting the env variable.
  4. PublishCodeCoverageResults@1: publish the report generated in step 2.

Option2:

  1. dotnet test: generate coverage file.
  2. PublishCodeCoverageResults@1: this task will generate report base on coverage xml file from step 1 and publish to ADO. But the report is generated by ReportGenerator 5.1.14.0

Option3:

  1. dotnet test: generate coverage file.
  2. PublishCodeCoverageResults@2: this can generate report and publish some report to code coverage tab, but using ADO layout.

Opeion 2 and 1 have no much different. But Option1 give your more customization option in ReportGenerator task. To me, Option 2 is easier to setup then option 1 and do the job, if you don't need too much customization on ReportGenerator task and dont mind to use ReportGenerator 5.1.14.0.

One thing is missing and I believe is useful is that the 'collapose all' and 'expand all' link. If I generate the report locally on my laptop, I can use them to quickly collapose the list. But they are not in ADO's coverage tab, meaning that I have to use scrol bar.

A-Rai-col commented 6 months ago

Thanks @jchenathub24 this clears my doubts about PublishCodeCoverageResults@2 ,I faced the same issue when using V2 task as it seems to use reportgenerator internally; First it generates the HtmlInline_AzurePipelines report and publishes that to the Code Coverage tab then after a couple of seconds it repopulates the tab with the default ADO layout.

I'm having to resort to using PublishCodeCoverageResults@1 as mentioned in your Option1 cause the ReportGenerator report is just so much better. Also wish there was collapse all & filter options in the report.

danielpalme commented 6 months ago

Also wish there was collapse all & filter options in the report.

That's because Azure DevOps blocks all JavaScript. If you download the full report artifact, those features will become available.

cremor commented 4 months ago

@danielpalme I think this issue should be reopened because the ReportGenerator documentation regarding Azure DevOps tasks should be updated. A few weeks ago Azure DevOps pipeline builds that use V1 of the PublishCodeCoverageResults tasks started showing deprecation warnings. See also the announcement here: https://devblogs.microsoft.com/devops/new-pccr-task/

Important part in that announcement:

Q: I’m using Azure DevOps Server version 2022. What will happen to me? A: In a future version of the Azure DevOps Server, we will also remove the V1 version of the publish code coverage results task. There will be a transitioning period where the V1 version will still run. During the transition period you’ll need to move from the V1 version to the V2 version, as when the transition period ends the V1 version will be removed and any pipeline containing that task will fail.

So there should be a documentation that shows how to correctly use ReportGenerator with PublishCodeCoverageResults V2.

I think if you don't care how the HTML report looks and if you don't need to set any additional ReportGenerator settings, then you can simply use PublishCodeCoverageResults@2 now, without using reportgenerator@5 at all, because ReportGenerator will still be used to read/convert and merge the coverage reports.

danielpalme commented 4 months ago

@cremor I updated the documentation here: https://github.com/danielpalme/ReportGenerator/wiki/Integration#attention


I think if you don't care how the HTML report looks and if you don't need to set any additional ReportGenerator settings, then you can simply use PublishCodeCoverageResults@2 now, without using reportgenerator@5 at all, because ReportGenerator will still be used to read/convert and merge the coverage reports.

The new version has several disadvantages regarding the report in the Code Coverage tab within Azure DevOps

danielpalme commented 3 months ago

I just released version 5.3.0 of ReportGenerator.

The Azure DevOps task now has a new setting publishCodeCoverageResults. This allows to publish the report in 'Code Coverage' tab directly. If set to true, it makes the 'PublishCodeCoverageResults' task(s) obsolete. They can be removed from your pipeline.

Example:

- task: reportgenerator@5
  displayName: ReportGenerator
  inputs:
    reports: 'coverage.xml'
    targetdir: 'coveragereport'
    publishCodeCoverageResults: true
MTomBosch commented 3 months ago

When publishing the report "your way" is then still the html file regenerated from azure devops (which does not contain the branch and method coverage and details per class) or is your richer report uploaded?

danielpalme commented 3 months ago

@MTomBosch It's the richer report (Sample report)

cremor commented 3 months ago

The Azure DevOps task now has a new setting publishCodeCoverageResults. This allows to publish the report in 'Code Coverage' tab directly. If set to true, it makes the 'PublishCodeCoverageResults' task(s) obsolete. They can be removed from your pipeline.

Thanks! Does this also work with code coverage line indicators in pull requests? I'd test it myself but TBH the feature seems to be flaky even with the official tasks 😅

Documentation Example when it works: grafik

danielpalme commented 3 months ago

@cremor

Does this also work with code coverage line indicators in pull requests?

I don't know. I have not tested this. Would like to hear if this also appears in PRs.

A-Rai-col commented 3 months ago

The Azure DevOps task now has a new setting publishCodeCoverageResults. This allows to publish the report in 'Code Coverage' tab directly. If set to true, it makes the 'PublishCodeCoverageResults' task(s) obsolete. They can be removed from your pipeline.

Thanks! Does this also work with code coverage line indicators in pull requests? I'd test it myself but TBH the feature seems to be flaky even with the official tasks 😅

Example when it works: grafik

Interesting. In the ms docs

Which coverage tools and result formats can be used for validating code coverage in pull requests? Code coverage for pull requests capability is currently only available for Visual Studio code coverage (.coverage) formats. This can be used if you publish code coverage using the Visual Studio Test task, the test verb of dotnet core task and the TRX option of the publish test results task. Support for other coverage tools and result formats will be added in future milestones.

They say it only supports .coverage formats though, so this will probably only work if you use the VSTest task.

cremor commented 3 months ago

They say it only supports .coverage formats though

That's wrong or outdated. The screenshot I posted above is from a project that doesn't do anything with a .coverage file. At least not explicitly. Here is the pipeline the project currently uses:

  - task: DotNetCoreCLI@2
    displayName: 'Run tests'
    inputs:
      command: 'test'
      # Collect code coverage in OpenCover format because this is what SonarCloud supports.
      arguments: '--no-restore --configuration $(buildConfiguration) --collect:"XPlat Code Coverage;Format=opencover"'
      publishTestResults: true

  - task: PublishCodeCoverageResults@2
    displayName: 'Publish code coverage report'
    inputs:
      summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.opencover.xml'

I'm planning to replace PublishCodeCoverageResults@2 with reportgenerator@5.

A-Rai-col commented 3 months ago 
  - task: PublishCodeCoverageResults@2
    displayName: 'Publish code coverage report'
    inputs:
      summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.opencover.xml'

I see, this might be an undisclosed feature they're adding the PublishCodeCoverageResults@2 task. 💭 As neither the V2 task docs nor the release notes for it, has any mention of code coverage for pull requests.

cremor commented 3 months ago

PublishCodeCoverageResults@2 uses https://github.com/microsoft/azure-pipelines-coveragepublisher (which uses ReportGenerator.Core). reportgenerator@5 uses the same publishing method as PublishCodeCoverageResults@1 used. I couldn't figure out where the source code of that is.

cremor commented 3 months ago

I don't know. I have not tested this. Would like to hear if this also appears in PRs.

I can now answer this. Sadly PR line coverage does not show up with reportgenerator@5 while it does work with PublishCodeCoverageResults@2. I don't know how PublishCodeCoverageResults@2 does it but I assume it is something in https://github.com/microsoft/azure-pipelines-coveragepublisher

At least I know that it is not related to the files which are published as artifacts. Because currently PublishCodeCoverageResults@2 doesn't publish any artifacts at all. Although that might be a bug, see microsoft/azure-pipelines-tasks#19547. Along with many other active issues for PublishCodeCoverageResults@2. It seems to be quite bad 😁

A-Rai-col commented 3 months ago

I can now answer this. Sadly PR line coverage does not show up with reportgenerator@5 while it does work with PublishCodeCoverageResults@2.

Still, cheers for sharing that something like code coverage for pull requests was a thing. That would be a very useful feature indeed. Hopefully, Microsoft eventually lets us publish ReportGenerator reports using PublishCodeCoverageResults@2 task which might have that feature built in.

Until then, thanks to @danielpalme we can now get rid of the deprecated warning that came with the V1 task. 🙌

jasells commented 2 months ago

OK, I just got caught up on this thread after finally getting the time to address the MS Publish code coverage results@v1 deprecation.

@danielpalme Been using this tool for year, such a great tool.

TY for adding the simple publish option in the latest of your ADO task, this solves the immediate issue of MS's deprecated publish coverage results task as well as the long-standing complicated config to publish the reports from your tool instead of the ADO-native reports.

Nothing like a forcing function to push progress!

To summarize this thread, this post shows how to publish this tool's reports directly to the Tests/Coverage tab in ADO without another "publish" task, MS or otherwise.

- task: reportgenerator@5
  displayName: ReportGenerator
  inputs:
    reports: 'coverage.xml'
    targetdir: 'coveragereport'
    publishCodeCoverageResults: true  << add this, remove MS' publish task V-any, no longer needed!

TY again!

@A-Rai-col

I still get line coverage after only changing the publish option and removing MS's publish task entirely. If it is missing, you may have your test step configured wrong.

I use this package to generate the raw coverlet data, which this tool can then consume to create the reports. https://www.nuget.org/packages/coverlet.msbuild/6.0.2#show-readme-container

Include it in all unit-test projects like:

 <PackageReference Include="coverlet.msbuild" Version="6.0.2" PrivateAssets="all">

The step to run tests in my pipeline:

 dotnet test [unitTestProjectPath] --configuration "$(BuildConfiguration)" --logger "trx;" /p:CollectCoverage=true /p:CoverletOutputFormat="cobertura%2copencover" /p:Exclude="[*Tests]*" /p:CopyLocalLockFileAssemblies=true
cremor commented 2 months ago

I still get line coverage after only changing the publish option and removing MS's publish task entirely. If it is missing, you may have your test step configured wrong.

@jasells My tests and coverage options are configured correctly. When I tested this a month ago, I simply created a pull request with a few changes and checked the PR for line coverage indicators. There were none when I used reportgenerator@5 with publishCodeCoverageResults: true. Then I simply changed reportgenerator@5 to PublishCodeCoverageResults@2 and the line coverage indicators showed up in the PR.

Are you sure that we are talking about the same thing? I explicitly mean the line coverage indicators in pull requests. I do not mean the coverage tab on the pipeline build result page.

Documentation Example when it works: grafik Green checkmarks show covered lines, red crosses show uncovered lines.

jasells commented 2 months ago

Are you sure that we are talking about the same thing? I explicitly mean the line coverage > indicators in pull requests. I do not mean the coverage tab on the pipeline build result page.

@cremor

Ah, I think we are talking about different things? I get the line-coverage in the coverage reports, not inline in the PR file-compare. I've never had that, didn't know it was an option. I'll look into that.

What I have done is save the coverage history to a Universal Package at end of pipeline, and download the latest before running the generator to get a coverage history graph. That might help you a little?

It would be nice to have to coverage inline in the PR, for sure.