Open pkuczynski opened 5 years ago
+1 on this issue!
This would be a great addition when working with a monorepo.
Is there a workaround for this?
Not that I know of :(
+1
I use jest-junit
and cobertura
reports both from front-end (create react app, jest) and back-end (.NET Core 3.0) but there is no merge option for both reports.
What a pity.
+1
Is there any update on this issue? I'm working on a monorepo and the fact that it cannot collect all the coverage reports limits the usage of this information in dashboards, for example. All the existing widgets to report code coverage in ADO dashboards rely on this data...
To get around this, we can merge all the files using ReportGenerator. This is what worked for me:
(Optional) If the code coverage is coming from different jobs, publish the coverage files as artifacts and then have a later job download all the artifacts
Create a report from all coverage files and publish
- task: reportgenerator@4
inputs:
reports: **/coverage.cobertura.xml;artifact/**/cobertura-coverage.xml
targetdir: $(Pipeline.Workspace)/coveragereport
This has worked well for me.
Work around is good and all, but in some orgs, it can take quite a while until you get a marketplace extension approved, if at all. 100% should be handled by Azure Pipelines and not some random extension.
fwiw it looks like this might have been fixed with the v2 of this task at https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/PublishCodeCoverageResultsV2/task.json with https://github.com/microsoft/azure-pipelines-tasks/pull/11144 but it hasn't been published to devops...not sure why was also asked in #13257
@MisinformedDNA where's that reportgenerator task come from?
Using reportgenerator in the way described above doesn't fully work for me. I get the following warning from the PublishCodeCoverageResults@1
task: [warning]No code coverage results were found to publish.
.
However, if i set summaryFileLocation: '$(Build.SourcesDirectory)/**/coverage.cobertura.xml'
then everything works but I'm back to getting the misleading warning: Multiple file or directory matches were found. Using the first match...
.
My guess is that something either is in the wrong place or is looking in the wrong place.
Can you share your YAML?
I have a project with three different modules, two Java and one Python. all three modules generate Cobertura based code coverage reports. I am able to merge all three into one using @danielpalme "ReportGenerator" extension. able to apply gating condition using merged coverage report. Please refer this.
The reportgenerator
can merge the results, and then I had to specify the reporttypes
and include Cobertura
to make it work
- task: DotNetCoreCLI@2
displayName: 'Create code coverage report'
inputs:
command: custom
custom: tool
arguments: 'run reportgenerator -reports:$(Build.SourcesDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)\CodeCoverage -reporttypes:HtmlInline_AzurePipelines;Cobertura'
- task: PublishCodeCoverageResults@1
displayName: 'Publish code coverage report'
inputs:
codeCoverageTool: 'cobertura'
summaryFileLocation: '$(Build.SourcesDirectory)\CodeCoverage\*cobertura.xml'
The
reportgenerator
can merge the results, and then I had to specify thereporttypes
and includeCobertura
to make it work- task: DotNetCoreCLI@2 displayName: 'Create code coverage report' inputs: command: custom custom: tool arguments: 'run reportgenerator -reports:$(Build.SourcesDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)\CodeCoverage -reporttypes:HtmlInline_AzurePipelines;Cobertura' - task: PublishCodeCoverageResults@1 displayName: 'Publish code coverage report' inputs: codeCoverageTool: 'cobertura' summaryFileLocation: '$(Build.SourcesDirectory)\CodeCoverage\*cobertura.xml'
In my case, I had to add the reportDirectory
param as well
- task: PublishCodeCoverageResults@1
displayName: Publish code coverage results
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: "$(System.DefaultWorkingDirectory)/testresults/coverage/reports/Cobertura.xml"
reportDirectory: "$(System.DefaultWorkingDirectory)/testresults/coverage/reports/"
also add a job level variable to disable the publisher generation of coverage reports. so that the self generated reports will be uploaded instead
- job: Docker
displayName: Capture version, build/push docker image
continueOnError: false
variables:
disable.coverage.autogenerate: true
steps:
Hi there,
when I try to use the workaround above as in:
pool:
vmImage: ubuntu-20.04
variables:
disable.coverage.autogenerate: 'true' # for report generator
steps:
# more tasks here...
- task: DotNetCoreCLI@2
displayName: 'Install ReportGenerator'
inputs:
command: custom
custom: tool
arguments: 'install --global dotnet-reportgenerator-globaltool --version 4.8.12'
- task: DotNetCoreCLI@2
displayName: 'Merge code coverage files'
inputs:
command: custom
custom: tool
arguments: 'run reportgenerator -reports:$(Build.SourcesDirectory)/build_reports/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/build_reports/code_coverage -reporttypes:HtmlInline_AzurePipelines;Cobertura'
I get the error message
Cannot find a tool in the manifest file that has a command named 'reportgenerator'.
Any ideas what's wrong here? @mike-bagheri and @jack4it , how did you guys install reportgenerator
? I'd appreciate any advice. Thank you so much in advance.
edit: Corrected false
to true
in disable.coverage.autogenerate
which, however, is unrelated to my question.
We just use a script for that
- script: dotnet tool install --global dotnet-reportgenerator-globaltool
displayName: Install ReportGenerator
- script: reportgenerator -reports:$(Agent.TempDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/CodeCoverage -reporttypes:Cobertura
displayName: Create Code coverage report
Hi there,
when I try to use the workaround above as in:
pool: vmImage: ubuntu-20.04 variables: disable.coverage.autogenerate: 'false' # for report generator steps: # more tasks here... - task: DotNetCoreCLI@2 displayName: 'Install ReportGenerator' inputs: command: custom custom: tool arguments: 'install --global dotnet-reportgenerator-globaltool --version 4.8.12' - task: DotNetCoreCLI@2 displayName: 'Merge code coverage files' inputs: command: custom custom: tool arguments: 'run reportgenerator -reports:$(Build.SourcesDirectory)/build_reports/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/build_reports/code_coverage -reporttypes:HtmlInline_AzurePipelines;Cobertura'
I get the error message
Cannot find a tool in the manifest file that has a command named 'reportgenerator'.
Any ideas what's wrong here? @mike-bagheri and @jack4it , how did you guys install
reportgenerator
? I'd appreciate any advice. Thank you so much in advance.
From the extension in Azure DevOps, installed this
You may also need to add this
- task: DotNetCoreCLI@2
displayName: 'Create .NET local manifest'
inputs:
command: custom
custom: new
arguments: 'tool-manifest
There a two options:
dotnet new tool-manifest
dotnet tool install --global dotnet-reportgenerator-globaltool
reportgenerator -reports:...
Thank you @danielpalme , but when I do a
- task: DotNetCoreCLI@2
displayName: 'Install ReportGenerator'
inputs:
command: custom
custom: new
arguments: 'tool-manifest'
- task: DotNetCoreCLI@2
displayName: 'Install ReportGenerator'
inputs:
command: custom
custom: tool
arguments: 'install --global dotnet-reportgenerator-globaltool --version 4.8.12'
- task: DotNetCoreCLI@2
displayName: 'Merge code coverage files'
inputs:
command: custom
custom: tool
arguments: 'run reportgenerator -reports:$(Build.SourcesDirectory)/build_reports/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/build_reports/code_coverage -reporttypes:HtmlInline_AzurePipelines;Cobertura'
I get a different pipeline error saying
Cannot find a tool in the manifest file that has a command named 'reportgenerator'.
Any idea what's wrong here? Do I always have to call reportgenerator
instead of dotnet run reportgenerator
?
edit: Corrected code snippet above.
I will also go and try your AzureDevops Extension.
@svdHero I just tried it myself, the following works for me:
- task: DotNetCoreCLI@2
displayName: 'Install ReportGenerator'
inputs:
command: custom
custom: tool
arguments: 'install --global dotnet-reportgenerator-globaltool --version 4.8.12'
- script: reportgenerator "-reports:$(Build.SourcesDirectory)/build_reports/**/coverage.cobertura.xml"" -targetdir:$(Build.SourcesDirectory)/build_reports/code_coverage" -reporttypes:HtmlInline_AzurePipelines;Cobertura
displayName: Merge code coverage files
This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days
Bamp/not stale
@acesiddhu is there an update on this?
This is a pain point. We have a repo that has java, python and dotnet modules. And cumbersome to merge all of them to a single report.
We've now run into this as well. The doc at https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results?view=azure-devops clearly states that "Multiple summary files will be merged into a single report.", but that is patently untrue. At the least the doc should be fixed to reflect the reality thay this feature is not yet included. Finding this out the hard way is time-consuming.
@drdamour commented on Jan 13, 2021, 8:12 PM GMT+1:
fwiw it looks like this might have been fixed with the v2 of this task at https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/PublishCodeCoverageResultsV2/task.json with #11144 but it hasn't been published to devops...not sure why was also asked in #13257
1.5 year later and v2 is still not available. :(
Keeping the issue alive; as mentioned by others still no fix, and based on this comment https://github.com/microsoft/azure-pipelines-tasks/issues/15160#issuecomment-902632589 doesn't seem like there is a plan to merge in the v2 version of this task which does have a combiner?
The suggested workaround of using ReportGenerator (https://github.com/danielpalme/ReportGenerator) does work but is a bit of a hack in the sense that the last user of PublishCodeCoverageResults
is the winner. This can be difficult if you have a complicated pipeline that leverages templates and requires a significant amount of coordination.
Same here. Only 1 out of many tests projects is included into the results.
I tried adding an additional task mentioned on Stack Overflow:
- task: reportgenerator@4
displayName: 'Merge code coverage reports'
inputs:
reports: '$(Agent.TempDirectory)\TestResults\**\coverage.cobertura.xml'
targetdir: '$(Agent.TempDirectory)\TestResults\Merged\'
reporttypes: Cobertura
verbosity: Verbose
but reportgenerator
is not and can't be installed to my ADO project.
but
reportgenerator
is not and can't be installed to my ADO project.
You have to hit the "Get it free" button on this website: https://marketplace.visualstudio.com/items?itemName=Palmmedia.reportgenerator
Then the task should be available (you should use reportgenerator@5)
You have to hit the "Get it free" button on this website:
Unfortunately I can't, my company's policy prohibits installing third-party extensions. Technically there is a process but it's near-impossible to get through.
I'm trying to work it around by install it as a global tool and run directly. But so far it doesn't make any difference: all tasks succeed, build succeeds, code coverage report stays the same.
I have multiple test projects which I run simultaneously by passing *.Tests.csproj
. Is the resulting code coverage results file's name unique?
- task: DotNetCoreCLI@2
displayName: 'Run tests'
inputs:
command: test
projects: '**\*Tests.csproj'
arguments: '--no-build --no-restore --configuration ${{ parameters.configuration }} --settings $(RepoRoot)\.runsettings --blame --collect "Code coverage" --logger "trx" --results-directory $(Agent.TempDirectory)\TestResults\ /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Agent.TempDirectory)\TestResults\'
publishTestResults: false
- task: DotNetCoreCLI@2
displayName: 'Install reportgenerator'
inputs:
command: custom
custom: tool
arguments: 'install --global dotnet-reportgenerator-globaltool'
- task: PowerShell@2
displayName: 'Merge code coverage reports'
inputs:
targetType: inline
script: |
$source = '$(Agent.TempDirectory)\TestResults\**\coverage.cobertura.xml'
$target = '$(Agent.TempDirectory)\TestResults\Merged\'
& reportgenerator.exe -reports:$source "-targetdir:$target" "-reporttypes:HtmlInline_AzurePipelines;Cobertura"
- task: PublishCodeCoverageResults@1
displayName: 'Publish code coverage'
condition: always()
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(Agent.TempDirectory)\TestResults\Merged\**\*.xml'
Where's the mistake?
This should work
disable.coverage.autogenerate: 'true'
...
- task: PublishCodeCoverageResults@1
displayName: 'Publish code coverage'
condition: always()
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(Agent.TempDirectory)\TestResults\Merged\Cobertura.xml'
reportDirectory: '$(Agent.TempDirectory)\TestResults\Merged'
The environment variable "disable.coverage.autogenerate" is important, otherwise PublishCodeCoverageResults will regenerate the coverage HTML report.
hi @danielpalme, thank you for your help! unfortunately I'm still unable to get it working.
What worked: the format has changed and it looks much nicer:
However it still shows only the first test project's results.
My YAML:
- task: DotNetCoreCLI@2
displayName: 'Run tests'
inputs:
command: test
projects: '**\*Tests.csproj'
arguments: '--no-build --no-restore --configuration ${{ parameters.configuration }} --settings $(RepoRoot)\.runsettings --blame --collect "Code coverage" --logger "trx" --results-directory $(Agent.TempDirectory)\TestResults\ /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Agent.TempDirectory)\TestResults\'
publishTestResults: false
- task: DotNetCoreCLI@2
displayName: 'Install reportgenerator'
inputs:
command: custom
custom: tool
arguments: 'install --global dotnet-reportgenerator-globaltool'
- task: PowerShell@2
displayName: 'Merge code coverage reports'
inputs:
targetType: inline
script: |
Write-Host '##vso[task.setVariable variable=disable.coverage.autogenerate;isSecret=false]true'
$source = '$(Agent.TempDirectory)\TestResults\**\coverage.cobertura.xml'
$target = '$(Agent.TempDirectory)\TestResults\Merged\'
& reportgenerator.exe -reports:$source -targetdir:$target -reporttypes:"HtmlInline_AzurePipelines;Cobertura"
- task: PublishCodeCoverageResults@1
displayName: 'Publish code coverage'
condition: always()
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(Agent.TempDirectory)\TestResults\Merged\**\*.xml'
reportDirectory: '$(Agent.TempDirectory)\TestResults\Merged'
Can you please take a look and see if I've missed anything?
@abatishchev You beat me to the punch I was just going to reply to say that you can install the tool manually via Dotnet-Tool and merge it, you've got it and your solution looks more or less like what we use internally. However as I mentioned above:
The suggested workaround of using ReportGenerator (https://github.com/danielpalme/ReportGenerator) does work but is a bit of a hack in the sense that the last user of PublishCodeCoverageResults is the winner. This can be difficult if you have a complicated pipeline that leverages templates and requires a significant amount of coordination.
Be careful!
Can you please take a look and see if I've missed anything?
I hope. Could you please add the following argument to reportgenerator.exe: -verbosity:verbose
Then you could check the output of that task.
For every coverage file that gets processed the following line should appear in the messages:
Loading report '%%PATH%%' x/y in memory
Do several coverage files get processed?
PublishCodeCoverageResults
@danielpalme
Not good :(
2023-01-17T20:53:46: Loading report 'C:\__w\_temp\TestResults\coverage.cobertura.xml' 1/1 in memory
Here's the "Run tests" task logs:
Attachments:
C:\__w\_temp\TestResults\d656b065-66c2-4f1d-a967-7ec1cda8404a\ContainerAdministrator_C13A536809CD_2023-01-17.20_45_39.coverage
Passed! - Failed: 0, Passed: 82, Skipped: 0, Total: 82, Duration: 27 s - Common.Tools.Tests.dll (net6.0)
Calculating coverage result...
Generating report 'C:\__w\_temp\TestResults\coverage.cobertura.xml'
Apparently, it's the "last run wins": the same file is overwritten by each consequent test project.
In most of my projects I use the package coverlet.collector
Then I run the tests with the following command (only important arguments added here): dotnet test --collect:"XPlat Code Coverage"
This results in cobertura file in each test project in a subfolder named TestResults\%%GUID%%\coverage.cobertura.xml
Then I pass all cobertura files to ReportGenerator with the following argument:
-reports:*Tests\TestResults\*\coverage.cobertura.xml
I've changed the "run tests" to this:
- task: DotNetCoreCLI@2
displayName: 'Run tests'
inputs:
command: test
projects: '**\*Tests.csproj'
arguments: '--no-build --no-restore --configuration ${{ parameters.configuration }} --settings $(RepoRoot)\.runsettings --blame --collect "XPlat Code Coverage" --logger "trx" --results-directory $(Agent.TempDirectory)\TestResults\ /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Agent.TempDirectory)\TestResults\'
publishTestResults: false
i.e. only from --collect "Code coverage"
to --collect "XPlat Code Coverage"
and it made no difference:
Results File: C:\__w\_temp\TestResults\ContainerAdministrator_AED0B744811A_2023-01-17_23_46_43.trx
Passed! - Failed: 0, Passed: 2, Skipped: 0, Total: 2, Duration: 81 ms - ConfigurationTests.dll (net6.0)
Calculating coverage result...
Generating report 'C:\__w\_temp\TestResults\coverage.net6.0.cobertura.xml'
Are you sure that the "Run tests" task executes all test projects? I'm not familiar with all the inputs of that task and your supplied arguments.
Yes, the Tests tab shows all 551 have been executed every time. The problem is in the code coverage file being overwritten every time.
How come yours gets a guid appended to its path?
Maybe you could try to remove all these arguments. I guess they are affecting the target directory. I'm using the setup described here: https://reportgenerator.io/getstarted#net
Instead of arguments: '--no-build --no-restore --configuration ${{ parameters.configuration }} --settings $(RepoRoot)\.runsettings --blame --collect "XPlat Code Coverage" --logger "trx" --results-directory $(Agent.TempDirectory)\TestResults\ /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Agent.TempDirectory)\TestResults\'
You could start with:
arguments: '--no-build --no-restore --configuration ${{ parameters.configuration }} --settings $(RepoRoot)\.runsettings --blame --collect "XPlat Code Coverage"'
hi @danielpalme, sorry - just coming back to this issue.
What doesn't work for me: changing it to "XPlat Code Coverage":
Data collection : Unable to find a datacollector with friendly name 'XPlat Code Coverage'.
Data collection : Could not find data collector 'XPlat Code Coverage'
However what seems to make a difference:
from
<DebugType>pdbonly</DebugType>
to
<DebugType>full</DebugType>
Hi,
is there any ETA or priority for this issue? We use 3rdParty generator as workaround, but it would be nice to have stable out-of-the-box solution from Microsoft. Thanks!
PublishCodeCoverageResults@2
works for me without any additional report merging
The report is waay simpler though.
The old one:
vs the new one:
Finally, all my years of tracking this issue have paid off. v2 is showing up in Azure DevOps for me as well and is listed in their documentation as a task, as of 7/10/2023.
It only merge files before publishing? If I have multiple jobs it means it won't work? For example, PublishTestResults works fine with multiple jobs if you set mergeTestResults=true
This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days
Required Information
Type: bug Enter Task Name: PublishCodeCoverageResults@1
Environment
Issue Description
Task PublishCodeCoverageResults@1 supports reports from multiple files, but the summary just shows the first match.
Task logs
The behavior for a task configured with a wildcard path for the coverage file is a little inconsistent:
In this example, there are few coverage.cobertura.xml files from separate test runs. Unfortunately, I receive warning: