search

MicrosoftPremier / VstsExtensions

Documentation and issue tracking for Microsoft Premier Services Visual Studio Team Services Extensions
MIT License
56 stars 14 forks source link

Code Coverage check adds the line from every run breaking the baseline #205

Closed bkqc closed 1 year ago

bkqc commented 1 year ago

Describe the context

Describe the problem and expected behavior In our current build, we have some tests that don't always succeed. We are working on fixing this but in the meantime, we often have to rerun the build using the button in the failed build summary (in french, it is labeled "Réexécuter les travaux non réussis"). The problem is each time we rerun, the blocks count from every run is included making an automatic failure for subsequent builds since the value compared is the block count and not the percentage.

  1. For instance, in one case, I got on my first run [ERROR] The code coverage (absolute) value (84.114%, 58397 blocks) is lower than the value of the previous build! The previous build had a coverage value of 84.3215% (233449 blocks).

  2. Second run was [ERROR] The code coverage (absolute) value (84.084%, 116363 blocks) is lower than the value of the previous build! The previous build had a coverage value of 84.3215% (233449 blocks).

  3. Third (not sure why it is still only 2 times the original): [ERROR] The code coverage (absolute) value (84.1155%, 116796 blocks) is lower than the value of the previous build! The previous build had a coverage value of 84.3215% (233449 blocks).

  4. Fourth: [ERROR] The code coverage (absolute) value (84.1159%, 175195 blocks) is lower than the value of the previous build! The previous build had a coverage value of 84.3215% (233449 blocks).

  5. Fifth (again, it seems it didn't add the previous run): [ERROR] The code coverage (absolute) value (84.1164%, 175196 blocks) is lower than the value of the previous build! The previous build had a coverage value of 84.3215% (233449 blocks).

  6. Sixth (succeeded): [SUCCESS] Code coverage policy passed with 84.1169% (233596/277704 blocks).

And now the base block count has increased again for the next builds.

ReneSchumacher commented 1 year ago

Hi @bkqc,

can you give a little more information about what technology your using and what type of code coverage you are generating? I would like to repro this and then send this feedback to the Azure DevOps product group.

This is not an issue with *Build Quality Checks" but with the way Azure DevOps Processes coverage data. I wasn't aware of it (and I'm not sure if this is intended behavior), but it seems that when you rerun a pipeline instead of running a new pipeline, code coverage data is published to the same result store as the previous runs. This would kick off the coverage merge job in the backend, which aggregates coverage results from multiple runs into one result object. Thus, it keeps adding the same coverage results over and over.

I don't understand why this only happens every other time, but I will investigate and collaborate with the product teams. My personal opinion is that this should not happen, but I guess this is the same for all summary data (warnings etc.).

Currently, the only workaround I can think of is running a new pipeline instead of rerunning it.

bkqc commented 1 year ago

The test projects are configured to use MsTest.TestAdapter and MsTest.TestFramework 2.2.10 and are built against .Net Framework 4.8 (though most tested projects are using .Net Framework 4.5.*). The solution is built using the Build Visual Studio v1.166.2 pipeline task. The tests task is Visual Studio Test v2.170.1 In the .runsettings file, we have the following line. I'm not sure what its precise role is though... <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">

If you need anything more, feel free to ask. Thank you for considering the request.

ReneSchumacher commented 1 year ago

Hi @bkqc,

I have now tried several ways to reproduce your issue but wasn't able to do so. I explicitly used Azure DevOps Server 2020.1.1 with agent version 2.181.2, but no matter what I do, the build summary and BQC is always showing the data from the latest run. Even if I create tests that randomly create different coverage, I can see the coverage change in each run.

There seems to be something special in your case. Can you share your full pipeline and maybe some of your test code with me? You can send the data to PSGerExtSupport@microsoft.com (just make sure to remove any sensitive data before sending it over). I hope I'll spot the difference between your environment and my repro environment then.

bkqc commented 1 year ago

Hello Rene, The pipeline is a classic pipeline. I got the definition json by looking at the history tab. Hope that's a good way of getting it. As for the test code, I'm not sure which tests to choose. Do you think some may be more subject than others to cause the problem.

PS Pipeline comments are in French. Hope this won't cause you too much trouble ;)

bkqc commented 1 year ago

Hello Rene, simply wondering if you received the email?

ReneSchumacher commented 1 year ago

Hey @bkqc,

sorry for not getting back to you earlier. I'm still trying to reproduce the issue but I was sick last week (too much Cologne carnival, I guess). I'll let you know (probably tomorrow) about the outcome of my tests.

ReneSchumacher commented 1 year ago

Hi again,

I don't see anything special in your pipeline that would explain the behavior. I can see that you're using a runsettings file, though. Can you send that via email as well? Maybe there's some setting that impacts the behavior.

ReneSchumacher commented 1 year ago

Hi @bkqc,

I was unable to find any issue with our task. Since the coverage values vary dramatically and are not just multiples, the root cause is likely the way you execute your tests. BQC only works if all tests are executed on every run. Otherwise, coverage will - naturally - be lower then it should be. I'm closing this issue now as it does not seem to be caused by our task. If you need additional assistance, please feel free to comment on the closed issue or open a new one.