Check Build Quality complains of code coverage when coverage increased!

[SoSain]

Hi We are using extension check build quality in our pipelines to check that code coverage % has not gone down.

What we observed recently is that the last build showed a code coverage of 11.42% in Summary view of the pipeline and 9.568% in the check build quality job

https://microsoft.visualstudio.com/Windows%20Defender/_build/results?buildId=31959364&view=results https://microsoft.visualstudio.com/Windows%20Defender/_build/results?buildId=31959364&view=logs&j=275f1d19-1bd8-5591-b06b-07d489ea915a&t=974ad5bd-9efb-5f41-bf5d-aaf7dd00193a

But next build which was having 11.90% coverage and in check build Quality job it showed 9.886% failed with message "The code coverage (percentage) value (9.8865%, 6681 blocks) is lower than the value of the previous build! The previous build had a coverage value of 10.9683% (7405 blocks)"

https://microsoft.visualstudio.com/Windows%20Defender/_build/results?buildId=31961005&view=results https://microsoft.visualstudio.com/Windows%20Defender/_build/results?buildId=31961005&view=logs&j=275f1d19-1bd8-5591-b06b-07d489ea915a&t=974ad5bd-9efb-5f41-bf5d-aaf7dd00193a

Can you please help why this is the issue as it seems next build actually inctreased the code coverage. Let me know what details more you want. Sharing few pics also that will help here

[ReneSchumacher]

Hi @SoSain,

it seems that I don't have permissions to access the Windows Defender project (I see all kinds of other projects in the Microsoft organization but not yours) so I cannot check the logs. The different coverage values shown in BQC and the build summary can be caused by multiple things:

1. You might have applied a filter to the coverage data in BQC so it is only evaluating a subset of the coverage.
2. You are not using line coverage in BQC. The build summary always shows line coverage.

Could you download the .coverage file from your test run and open it in Visual Studio to check the coverage values? The block coverage value in that file should match the value from BQC.

[SoSain]

Hi René

Thank you for your prompt reply.

I downloaded the .coverage file from my test run in Visual Studio and checked the coverage value. That for Build no. "31994099" here Pipelines - Run 31994099 (visualstudio.com)https://microsoft.visualstudio.com/Windows%20Defender/_build/results?buildId=31994099&view=codecoverage-tab shows like below

[Graphical user interface, text, application Description automatically generated]

As you can see the Block coverage value above 9.89%. My concern is that the immediate previous build run on this branch here Pipelines - Run 31959364 (visualstudio.com)https://microsoft.visualstudio.com/Windows%20Defender/_build/results?buildId=31959364&view=codecoverage-tab, had a coverage of 9.57% [Graphical user interface Description automatically generated with medium confidence]

So you can see we actually improved from 9.57% to 9.89% but still the build no 31994099, failed with message below which is not clear. This is what we want to understand why the BQC is finding a value of 11.01% and failing next build when in last build the value was 9.57% and this build 9.89% [Graphical user interface, text, application Description automatically generated]

Regards, SourAV.

Note: Am sending this mail as its convenient for me to send it at this time. Feel free to reply, take action, when its convenient for you!

From: René Schumacher notifications@github.com Sent: Thursday, March 4, 2021 5:27 PM To: MicrosoftPremier/VstsExtensions VstsExtensions@noreply.github.com Cc: Sourav Sain sosain@microsoft.com; Mention mention@noreply.github.com Subject: Re: [MicrosoftPremier/VstsExtensions] Check Build Quality complains of code coverage when coverage increased! (#136)

Hi @SoSainhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSoSain&data=04%7C01%7Csosain%40microsoft.com%7Cdb888fde5e7047407ea608d8df049af3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504558179136707%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XcY%2BXTolrf43nQ8h5sjZxqRhyj%2FUjAcBMv0Cb%2B7%2FjfE%3D&reserved=0,

it seems that I don't have permissions to access the Windows Defender project (I see all kinds of other projects in the Microsoft organization but not yours) so I cannot check the logs. The different coverage values shown in BQC and the build summary can be caused by multiple things:

1. You might have applied a filter to the coverage data in BQC so it is only evaluating a subset of the coverage.
2. You are not using line coverage in BQC. The build summary always shows line coverage.

Could you download the .coverage file from your test run and open it in Visual Studio to check the coverage values? The block coverage value in that file should match the value from BQC.

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftPremier%2FVstsExtensions%2Fissues%2F136%23issuecomment-790561130&data=04%7C01%7Csosain%40microsoft.com%7Cdb888fde5e7047407ea608d8df049af3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504558179146695%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xJsLdxAb6zNKchc8UeGAyQQcrkjBJLa0PIEJECnB1ZA%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKNSLIWUKPCWGQWIWZLFRYLTB5YQNANCNFSM4YTADVKQ&data=04%7C01%7Csosain%40microsoft.com%7Cdb888fde5e7047407ea608d8df049af3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504558179146695%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BbYgn0KEiHO6dB%2BdcHm7lf%2FPe2v3mNpGdaVvaj5SDbA%3D&reserved=0.

[ReneSchumacher]

Hi @SoSain,

apparently, replying via email removes the screenshots. Could you please post the screenshots directly here or send your information to PSGerExtSupport@microsoft.com? We can then work internally to look at the issue. If possible, could you give me permissions to access your pipeline directly (read access is sufficient)? My alias is reneschu.

[SoSain]

Hi René

Thank you for your prompt reply.

I downloaded the .coverage file from my test run in Visual Studio and checked the coverage value. That for Build no. “31994099” here Pipelines - Run 31994099 (visualstudio.com) shows like below

As you can see the Block coverage value above 9.89%. My concern is that the immediate previous build run on this branch here [Pipelines - Run 31959364 (visualstudio.com)](https://microsoft.visualstudio.com/Windows Defender/_build/results?buildId=31959364&view=codecoverage-tab), had a coverage of 9.57%

So you can see we actually improved from 9.57% to 9.89% but still the build no 31994099, failed with message below which is not clear. This is what we want to understand why the BQC is finding a value of 11.01% and failing next build when in last build the value was 9.57% and this build 9.89%

[ReneSchumacher]

Hi @SoSain,

thanks for the updated images. This is indeed strange. However, the number of blocks in the previous build as seen by BQC is totally different from the one in your coverage file. BQC has found 7442 covered blocks while the coverage file only has 6435 covered blocks. Are you sure that BQC is looking at the same base branch? It logs the baseline build ID to the task log so could you please check if it takes a different baseline build?

Just as a side note: BQC only looks as succeeded builds when choosing a baseline unless you tell it to include partially succeeded builds as well. It will never look at failed builds.

[ReneSchumacher]

Hi again,

any update for this issue?

[ReneSchumacher]

I'm closing this issue since there hasn't been any update for almost a month. If you still need help, feel free to post additional comments and I'll reopen it again.

[DenysGranevych]

~Also reproduced for my case.~ Nope, sorry. Just need to be careful. Dont forget to update value coverageFailOption: fixed if you dont want to compare with prev build

[ReneSchumacher]

Hi @DenysGranevych,

can you please send the logfiles for the currently failing and the previously succeeding build to PSGerExtSupport@microsoft.com, so we can take a look at them?

[DenysGranevych]

Hi @DenysGranevych,

can you please send the logfiles for the currently failing and the previously succeeding build to PSGerExtSupport@microsoft.com, so we can take a look at them?

sorry, it is my fail. All works good

[ReneSchumacher]

No harm done :-)