search

MicrosoftPremier / VstsExtensions

Documentation and issue tracking for Microsoft Premier Services Visual Studio Team Services Extensions
MIT License
59 stars 14 forks source link

Getting inconsistent coverage numbers #128

Closed geeknoid closed 3 years ago

geeknoid commented 3 years ago

We've been seeing inconsistent coverage numbers reported by the BQC task in our build. We've been using BQC for a while, but the problems emerged when we tried to increase the coverage threshold closer to what our current coverage level is, at which point BQC started complaining our coverage didn't meet the threshold. From run to run, the coverage number reported by BQC in its error message varies even though the actual code hasn't changed. I can just do "Rerun jobs" and the number changes.

The coverage number reported by Azure proper is consistent from run to run, and matches exactly the coverage number I get when I run the tests in Visual Studio on my machine. This number is consistently higher than what BQC reports in its error message.

In the image below, you can see the number Azure reports on the right side, while you see the error message reported by the BQC task, which shows a lower number. If I rerun the job, the BQC number in the error will be different.

image

geeknoid commented 3 years ago

Also note that the BQC message mentions 467 blocks, while in fact we have several thousand coverage blocks in our build.

ReneSchumacher commented 3 years ago

Hi @geeknoid,

This looks like a timing issue. As you already contacted me internally via email, I'm taking this offline and will only post the resolution here.

Thanks, René

ReneSchumacher commented 3 years ago

I believe this one has been resolved internally so I'm closing the issue.