Closed aolszowka closed 5 months ago
ReneSchumacher
commented
8 months ago Hi @aolszowka,
thanks for the very good repro steps. I'm going to check this on our side. The biggest difference between v1 and v2 of the publish task seems to be the coverage format. V2 creates a cjson coverage file (see here) and publishes it to Azure DevOps. I don't know what this should prevent the API from returning the correct values, but I'll check with the Azure DevOps testing tools team.
ReneSchumacher
commented
8 months ago First assumption: it seems like v2 of the publish task uploads the cjson file but does not trigger coverage processing in the backend. While the pipeline is running, the coverage API only returns an empty data set:
{
"coverageData": [],
"build": {
"id": "5010",
"url": "https://dev.azure.com/almtcger/_apis/build/Builds/5010"
},
"deltaBuild": null,
"coverageDetailedSummaryStatus": "codeCoverageSuccess"
}Coverage processing only starts after the pipeline run has finished, which is way too late for BQC to pick up the data. API results show that even after the build is finished it takes a couple seconds for the coverage job in the backend to run and then the API returns the proper results:
{
"coverageData": [
{
"coverageStats": [
{
"label": "Lines",
"position": 2,
"total": 53,
"covered": 52,
"isDeltaAvailable": false,
"delta": 0
}
],
"buildPlatform": "",
"buildFlavor": ""
}
],
"build": {
"id": "5010",
"url": "https://dev.azure.com/almtcger/_apis/build/Builds/5010"
},
"deltaBuild": null,
"status": "finalized",
"coverageDetailedSummaryStatus": "finalized"
}I will talk to the product group and see if they can trigger the job earlier. If not, there is currently no way to make BQC work with v2 of the publish coverage results task. We're also working on a version of the task that does the coverage processing itself, but I cannot say when this will be finished.
aolszowka
commented
8 months ago @ReneSchumacher Thank you for the quick turn around and Root Cause Analysis of this issue.
Is there any publicly posted tracker (IE GitHub Issue) that I can subscribe to, in order to track progress on the external report?
At the end of the day, we'll probably stick with the workaround; but I would like to document why we're stuck on a particular version and still using a workaround. Would it be worth filing documentation change requests as well in order to raise awareness of the issue for other DevOps Engineers attempting to use this combination?
ReneSchumacher
commented
8 months ago Currently, there's no such tracker. We could create something for the task, but some of the work the PG does is not tracked publicly on GitHub but on our internal Azure DevOps project. Thus, I'm not sure if they will update any issue for the task. Nevertheless, I can create an issue in the azure-pipelines-tasks repo, reference this one, and update the issues when I receive more information internally.
ReneSchumacher
commented
8 months ago @aolszowka I have updated an existing issue in the azure-pipelines-tasks repo with a reference to this one. I'll point the PG to both issues.
ReneSchumacher
commented
5 months ago I have received an update from PG regarding the compatibility of PublishCodeCoverageResults@2 and BQC. They have deployed a fixed version (2.231.0) of the task to the cloud service. This works with BQC, but V2 of the task only supports line coverage.
Thus, if you are using line coverage, you can now safely switch to v2 of the publish code coverage results task. If you need other coverage types like branch coverage, you must still use V2 of the publish code coverage results task. PG is also in the process of updating our docs to include this change.
Describe the context
Describe the problem and expected behavior After upgrading to
PublishCodeCoverageResults@2theBuildQualityChecks@9Spins with the following output:We believe this is the root cause of this persons issue as well; which was previously reported: https://github.com/MicrosoftPremier/VstsExtensions/issues/220 this user was able to work around it because they downgraded to
PublishCodeCoverageResults@1.We're seeing it now after upgrading to
PublishCodeCoverageResults@2; we did this because we wanted to use the ability of the new task to combine multiple Cobertura results (See https://github.com/microsoft/azure-pipelines-tasks/issues/10353) without having to use the work around ofReportGenerator(https://github.com/danielpalme/ReportGenerator) to perform the combination ourselves.However, in doing this we encountered the above.
Consider the following toy project layout (a zipped version is attached to this report and will also be emailed to the address provided) CodeCoverageCheck.zip:
Project Layout
azurepipeline.yml
CodeCoverageCheck.sln
CodeCoverageCheck.csproj
CodeCoverageToy.cs
CodeCoverageCheck.Tests.csproj
CodeCoverageToyTests.cs
CodeCoverageCheck.Tests2.csproj
CodeCoverageToy2Tests.cs
CodeCoverageCheck2.csproj
CodeCoverageToy2.cs
When running this in the pipeline we do see that coverage is reported (as seen in the screenshot below); however for some reason this extension does not pick up the report:
If you look at the output of the
publish code coverage (dotnet)task you don't see the summary like you will in the work around below:Work Around
While not ideal, you can fall back to the legacy
PublishCodeCoverageResults@1, however doing so requires you to convert back to usingReportGeneratorto pick up the multiple reports, an example of this working is here:azurepipeline.yml
This again is less than ideal; but when ran this way this works; differences seem to come in the publish task (in this version renamed to
publish code coverage (combined)):With this the reporter seems to be happy (obviously complaining about a lack of baseline):
However from the GUI perspective there's no way to tell that this is any different:
Task logs I have held off providing these in lieu of providing the entire reproducing project above. Should you need these logs I would be happy to provide them.