The BuildQualityGate check doesn't seem to be able to compare line coverage from different report types

[sjd2021]

The BuildQualityGate extension has been very helpful. For PRs, I use Cobertura so people can see what the opportunities for improvement are. I have to use the merge report tool, etc., unfortunately. So in the target branch for commits, I use the regular --collect "Code Coverage" option in the dotnet test command since it allows me to simplify that pipeline because it can publish the code coverage percentage for comparison purposes without the extra 3 steps. When the BuildQualityGate checks the base reference, it finds almost 2x as many lines in that coverage report than what it found when using the merged result of the cobertura files.

Is there any way to use this extension to compare across two different report types? I'm aware that the issues I'm having may transcend this extension, but I'm not sure where else to ask this. I just don't want the extra overhead in the target branch's pipeline since I do need the coverage % reported for a baseline, but I do not need the extra steps or artifacts which appear to be requirements for using cobertura.

[ReneSchumacher]

Hi @sjd2021,

I think I'll need more information to fully grasp the problem here. First, I don't really understand why you need two different ways of reporting coverage. What prevents you from using the regular --collect parameter for PRs as well?

Second, BQC does not really care how the coverage data has been generated since we don't evaluate/parse the coverage data in the task. Instead, coverage values are read from Azure DevOps using the code coverage APIs. So, it shouldn't really make any difference when comparing data from Cobertura with data from another coverage tool.

I can only think of two causes for your issue: either the coverage tools really calculate different values, or you have multiple coverage datasets that are merged by Azure DevOps and result in aggregated values. To figure out what's going on, could you please send me your pipeline definitions for both the builds running on the target branch and builds running for PRs? In addition, could you please set the variables System.Debug and BQC.LogRawData for your pipelines to true and send me the logs for the BQC task for both pipelines (target branch and PR)?

Please send the information to PSGerExtSupport@microsoft.com. I'm pretty sure we can get this sorted out 😉

[sjd2021]

Hi @ReneSchumacher
Thanks for responding so quickly!

I am using XPlat on PRs because I want developers who do not have VS enterprise to be able to see, quickly, the coverage report and identify areas of possible improvement. In the target branch, I don't really care about that because everything went through a PR before it got there unless there was a hotfix (very rare and has not happened yet) in which case we don't care about the details. I also don't want the big ol' coverage report artifact having to be used as part of our releases which unfortunately download every artifact from the pipeline. The only way that I've found that I'm able to get the code coverage percentage reported without a separate artifact being created and downloaded is to use the standard --collect "Code Coverage" argument.

I swear they're both running the same commands (otherwise) against the same test projects 😅 Maybe there's confusion because I have two separate test projects, but I'm not sure why those values are getting reported to Azure devops differently. I'll send the logs right over.

[ReneSchumacher]

Hi @sjd2021,

I still haven't received any logs. Did you send them? And if so, can you check if there has been a problem while sending them?

[sjd2021]

@ReneSchumacher sorry, I haven't been in the office yet. I need time to revert the pipeline back to xml produce the logs again.

[ReneSchumacher]

Ah, no problem. I just wanted to make sure that you're not waiting for me and I'm waiting for you 😄

[ReneSchumacher]

Hi @sjd2021,

I lost track of this issue, sorry for that. However, I don't think I have received your logs yet. Did you resolve the issue in the meantime?

[sjd2021]

@ReneSchumacher I replied just now. Thanks again for your help and patience.

[sjd2021]

Is the code that demonstrates how the code coverage is read for a particular branch available anywhere? At this point, I'm thinking of just writing something that allows me to specify what the code coverage percentage is for this particular build in hopes that this extension will work. I just can't find a workable solution, so I figure I can parse the values from the output I have and use a REST endpoint to set the percentage myself.

[ReneSchumacher]

Hi @sjd2021,

sorry for the delay, I have been on vacation for the last couple weeks. The code used to read the data from Azure DevOps can be found in the following two files (had to add the .txt extension to allow me to attach them):

• CoverageValueReader.ts.txt
  This contains the actual logic for reading, filtering, and aggregating the coverage values. The important method is getCodeCoverageSummaryFromBuild. We're using the Azure DevOps Node API to read the data, which is done by the TestApiHelper below.
• TestApiHelper.ts.txt
  This actually reads the data and uses a waiter implementation to wait for the merge job in Azure DevOps. The important method here is getCodeCoverageData which just calls the getCodeCoverageSummary Azure DevOps API.

I don't think you'll have a chance to directly set coverage values in Azure DevOps. This is all done in the background through the coverage file publishing process. 😞

[ReneSchumacher]

Hi @sjd2021,

I haven't had much time to spend on our extensions in the past. Do you need more assistance with this? Imho, the coverage type should not play a big role for BQC, since coverage is read from Azure DevOps and the Azure DevOps backend takes care of evaluating all supported coverage types and transforming them into a unified coverage summary object. Thus, the data should be available and BQC should just work.

I'm closing this issue for now as it has been stale for a couple months. If you need more help, feel free to either comment on the closed issue or open a new one.