Code Coverage fails to pull covered lines down and always recognizes 0% as lines covered

[rdurham285]

When this step runs in our CI, we get a warning that states "[WARNING] Cannot read property 'id' of undefined" and it seems the check fails to retrieve the stored code coverage as it only recognizes 0% of the covered lines.

This started this morning with the new update to 7.6.1

Validating code coverage policy...

[debug]System.TeamProject=StrategicSleeves

[debug]Build.BuildId=10059

[debug]CoverageValueReader.getCoverageValue

[debug]Build.BuildId=10059

[debug]CoverageValueReader.getCodeCoverageSummaryFromBuild

[debug]getCodeCoverageData

[debug]prepareClients

[debug]PSGer.Cover.PollInterval=undefined

[debug]PSGer.Cover.MaxWaitTime=undefined

[debug]PSGer.Cover.StableDataCount=undefined

[WARNING] Cannot read property 'id' of undefined

[warning]Cannot read property 'id' of undefined

[debug]Processed: ##vso[task.logissue type=warning;]Cannot read property 'id' of undefined

[debug]CoverageValueReader.filterCoverageDataByConfigAndPlatform

[debug]buildConfiguration=null

[debug]buildPlatform=null

[debug]explicitFilter=false

[debug]Inspecting 0 code coverage data sets...

[debug]CoverageValueReader.aggregateCoverageValues

[debug]coverageType=lines

[debug]CoverageValueReader.calculateCoveragePercentageWithPrecision

[debug]coveragePrecision=4

[debug]coverageVariance=null

[debug]treat0of0as100=false

[debug]treat0of0as100=false

Total lines: 0 Covered lines: 0 Code Coverage (%): 0

[debug]coverageFailOption=fixed

[debug]FixedCoverageThresholdRule.isFulfilledImpl

[debug]coverageThreshold=15

[ERROR] The code coverage value (0%, 0 lines) is lower than the minimum value (15%)!

[error]The code coverage value (0%, 0 lines) is lower than the minimum value (15%)!

[ReneSchumacher]

Hi @rdurham285,

could you send the full task log to PSGerExtSupport@microsoft.com? I'm not sure how this issue could possibly be related to the change we made in v7.6.1, but the full log should have enough information for me to figure out what's going on.

Thanks, René

[jjmarty]

We also just started experiencing this issue today. We're still running on version 6.3.0

2020-10-22T16:27:47.0130030Z ##[section]Starting: Check build quality 2020-10-22T16:27:47.0318139Z ============================================================================== 2020-10-22T16:27:47.0318760Z Task : Build Quality Checks 2020-10-22T16:27:47.0319396Z Description : Breaks a build based on quality metrics like number of warnings or code coverage. 2020-10-22T16:27:47.0319933Z Version : 6.3.0 2020-10-22T16:27:47.0320410Z Author : Microsoft Premier Services 2020-10-22T16:27:47.0321134Z Help : [Docs] 2020-10-22T16:27:47.0321893Z ============================================================================== 2020-10-22T16:27:48.3992416Z SystemVssConnection exists true 2020-10-22T16:27:48.7862208Z Using IdentifierJobResolver 2020-10-22T16:27:48.8614468Z Validating code coverage policy... 2020-10-22T16:27:51.2564170Z [WARNING] Cannot read property 'id' of undefined 2020-10-22T16:27:51.2565930Z ##[warning]Cannot read property 'id' of undefined 2020-10-22T16:27:51.2587654Z Total blocks: 0 2020-10-22T16:27:51.2588064Z Covered blocks: 0 2020-10-22T16:27:51.2588442Z Code Coverage (%): 0 2020-10-22T16:27:51.6342928Z Found baseline build with ID 77994. 2020-10-22T16:27:51.9929292Z Successfully read code coverage data from build. 2020-10-22T16:27:51.9931313Z Evaluating coverage data from 1 filtered code coverage data sets... 2020-10-22T16:27:51.9932249Z Total blocks: 25217 2020-10-22T16:27:51.9932624Z Covered blocks: 23675 2020-10-22T16:27:51.9933280Z Code Coverage (%): 93.89 2020-10-22T16:27:51.9935455Z [ERROR] The code coverage value (0%, 0 blocks) is lower than the value of the previous build including variance! The previous build had a coverage value of 93.89% (23675 blocks). 2020-10-22T16:27:51.9937162Z ##[error]The code coverage value (0%, 0 blocks) is lower than the value of the previous build including variance! The previous build had a coverage value of 93.89% (23675 blocks). 2020-10-22T16:27:51.9965075Z ##[error]At least one build quality policy was violated. See Build Quality Checks section for more details. 2020-10-22T16:27:52.0053212Z ##[section]Finishing: Check build quality

[aaroncearleytcms]

same here:

2020-10-22T21:09:14.8309696Z ##[section]Starting: Check build quality 2020-10-22T21:09:14.8470798Z ============================================================================== 2020-10-22T21:09:14.8471182Z Task : Build Quality Checks 2020-10-22T21:09:14.8471526Z Description : Breaks a build based on quality metrics like number of warnings or code coverage. 2020-10-22T21:09:14.8471829Z Version : 6.3.0 2020-10-22T21:09:14.8472088Z Author : Microsoft Premier Services 2020-10-22T21:09:14.8472505Z Help : [Docs] 2020-10-22T21:09:14.8472937Z ============================================================================== 2020-10-22T21:09:15.4255353Z SystemVssConnection exists true 2020-10-22T21:09:15.8136141Z Using IdentifierJobResolver 2020-10-22T21:09:15.9044020Z Validating code coverage policy... 2020-10-22T21:09:18.8107143Z [WARNING] Cannot read property 'id' of undefined 2020-10-22T21:09:18.8108872Z ##[warning]Cannot read property 'id' of undefined 2020-10-22T21:09:18.8131138Z Total blocks: 0 2020-10-22T21:09:18.8131792Z Covered blocks: 0 2020-10-22T21:09:18.8132168Z Code Coverage (%): 0 2020-10-22T21:09:18.8168661Z [ERROR] The code coverage value (0%, 0 blocks) is lower than the minimum value (49%)! 2020-10-22T21:09:18.8169793Z ##[error]The code coverage value (0%, 0 blocks) is lower than the minimum value (49%)! 2020-10-22T21:09:18.8202640Z ##[error]At least one build quality policy was violated. See Build Quality Checks section for more details. 2020-10-22T21:09:18.8280555Z ##[section]Finishing: Check build quality

this is after the build and unit testing in the pipeline which run successfully, so whoever put that this is the result of an unsuccessful merge. this is not the case for me.

[ReneSchumacher]

Thanks all,

if it's affecting older versions, there has to be a change in the Azure DevOps API. I'll try to repro that internally. Could at least one of you please run the pipeline again with the variables System.Debug and BQC.LogRawData set to true and send the task log to PSGerExtSupport@microsoft.com?

[aaroncearleytcms]

ok. I am pulling the logs together for the task. I will email it once it is done.

[aaroncearleytcms]

ok. the email was sent to PSGerExtSupport@microsoft.com with the debug log. the title of the email is "RE: [MicrosoftPremier/VstsExtensions] Code Coverage fails to pull covered lines down and always recognizes 0% as lines covered (#121)." Let me know if you need any other information. I will try to update this ticket with any information I receive on this issue.

[ReneSchumacher]

Thanks @aaroncearleytcms!

Quick update for all of you: there has indeed been a breaking change in the Azure DevOps backend that results in a change of the return value from the code coverage API. Instead of returning empty code coverage like this

{"coverageData":[],"build":{"id":"2603","url":"https://dev.azure.com/{organization}/_apis/build/Builds/2603"},"deltaBuild":null}

Azure DevOps currently returns this instead

{"deltaBuild":null}

The missing build information is causing the null reference exception in the code. I've already contacted the Azure DevOps testing tools team to see how we can quickly fix this. Hopefully, we'll have a fix (or at least a workaround) tomorrow.

[davidezordan]

Thank you, @ReneSchumacher for the update. Started getting the same error today:

Starting: Check build quality
==============================================================================
Task         : Build Quality Checks
Description  : Breaks a build based on quality metrics like number of warnings or code coverage.
Version      : 7.6.1
Author       : Microsoft
Help         : [[Docs]](https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/overview.md)
==============================================================================
SystemVssConnection exists true
Using IdentifierJobResolver
Validating code coverage policy...
[WARNING] Cannot read property 'id' of undefined
##[warning]Cannot read property 'id' of undefined
Total lines: 0
Covered lines: 0
Code Coverage (%): 0
[ERROR] The code coverage value (0%, 0 lines) is lower than the minimum value (60%)!
##[error]The code coverage value (0%, 0 lines) is lower than the minimum value (60%)!
Finishing: Check build quality

[Anvas007]

Same issue here: CI failing with error:

[WARNING] Cannot read property 'id' of undefined

[warning]Cannot read property 'id' of undefined

Total blocks: 0 Covered blocks: 0 Code Coverage (%): 0 [ERROR] The code coverage value (0%, 0 blocks) is lower than the minimum value (25%)!

[error]The code coverage value (0%, 0 blocks) is lower than the minimum value (25%)!

[error]At least one build quality policy was violated. See Build Quality Checks section for more details.

[nishantms]

@Anvas007 / @rdurham285 Can you please share the full build logs instead of BQC task logs. you can also mail us to devops_tools@microsoft.com.

cc: @ReneSchumacher

[nishantms]

@rdurham285

can you help us with the repsonse for the api for which the issue is happening.

https://dev.azure.com/<ACCOUNT>/<PROJECT>/_apis/test/CodeCoverage?buildId=<BUILDID>

[anshii03]

@rdurham285 Also please share the test platform version for your build :

Also , please tell if you can download and open .coverage attachments from Tests tab and if those .coverage attachments are non-empty i.e. they should have coverage data ?

[aaroncearleytcms]

Hello Everyone,

I have further update. as @ReneSchumacher has mentioned, it is apparently a breaking change for the api that this uses. I have submitted further information for my issue and he is working on it. until then I have disabled the code coverage step so that our pipeline can keep going. let me know if anyone can think of a better mitigation of this issue.

[rdurham285]

For our .net builds, we are not using the Test Platform Installer. It's using VS Pro 2017 that is installed on our build machines.

The .coverage files are not empty though.

The experience is consistent across our stack. We have an angular code base using ng test to generate test results and we parse the code coverage output with Cobertura. These builds also are recognizing and rendering the code coverage % and reports.

Response for that API request is the same as others mentioned before:

{ "deltaBuild": null }

On Fri, Oct 23, 2020 at 8:46 AM anshii03 notifications@github.com wrote:

@rdurham285 https://github.com/rdurham285 Also please share the test platform version for your build :

[image: image] https://user-images.githubusercontent.com/9054199/97011143-ef8ef080-1563-11eb-9513-3b054e19390d.png

Also , please tell if you can download and open .coverage attachments from Tests tab and if those .coverage attachments are non-empty i.e. they should have coverage data ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MicrosoftPremier/VstsExtensions/issues/121#issuecomment-715351278, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUHMGMU5N6RJFWRNYVO223SMGCKVANCNFSM4S3WAY3A .

[jjardina]

My company is having this exact same issue and our builds keep failing because of this bug. My lead developer asked to open a critical ticket with Microsoft. Do you think that would help get faster resolution to this issue or would it be a waste of money and everything that can be done is being done at this point?

[aaroncearleytcms]

Hello Everyone,

I just talked to @ReneSchumacher 10 minutes ago and he mentioned he is looking into a workaround. They are presently looking into this. They might escalate it based on the number of people but I also do not want to overwhelm them either. So to answer your question directly, I am not sure. I will leave whether you want to submit an additional issue up to your discretion.

[jjardina]

@ReneSchumacher Opened Case 120102324006130 with Microsoft concerning this issue.

[ReneSchumacher]

Thanks everyone for sending over information and letting us know that you're affected by the issue. The Azure DevOps product teams are already investigating the issue. So far, we couldn't reproduce it on our organizations. Thus, the issue is most probably caused by a rollout to a specific deployment ring.

Could you please let us know the region your Azure DevOps organization is in? In addition, it would be very helpful if someone could send over the full pipeline logs for one of the failing pipelines. You can send them either to PSGerExtSupport@microsoft.com or devops_tools@microsoft.com. This will greatly help us with the investigation.

I'm currently preparing the release of a new version of BQC that should have a workaround for the issue. Since I don't have a repro environment, though, you might see other issues showing up. The new version should be out in a couple minutes. Please post any new issue here as well and send in the debug log from the BQC task, so I can iterate over the solution if necessary.

[jjardina]

@ReneSchumacher My Azure DevOps presence is in Central US I will send the complete build log to the email you provided.

[aaroncearleytcms]

for those interested in finding the region setting, here is a url to describe where to find it. https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/change-organization-location?view=azure-devops

[ReneSchumacher]

Alright, deployment to the marketplace of v7.6.2 of the BQC extension is running. You should see the new version in about 10 minutes. Please let me know if the workaround helps.

[aaroncearleytcms]

I have sent the complete build log to @ReneSchumacher . @jjardina I am sure sending another log will help.

[crystalgeek]

@ReneSchumacher - I'm testing this version and seeing the following on repeat for the last 5 minutes

Waiting for code coverage data...

##[debug]-------------------- RAW DATA --------------------
##[debug]buildCoverageSummary:
##[debug]{"deltaBuild":null}
##[debug]------------------ END RAW DATA ------------------

Waiting for code coverage data...

[ReneSchumacher]

Thanks, @woehrl01

All: If BQC v7.6.2 times out now, please send us the name of your Azure DevOps organization. We have one more option that might mitigate the issue.

[jjardina]

@ReneSchumacher Mine seems to be timing out as well. Sending you an email with data from last test and Azure DevOps organization name.

[nishantms]

Hi All, we understood the problem and root caused the issue. We will be deploying hot fix in few days. Will update once done.

[jjardina]

@ReneSchumacher and @nishantms Ran my jobs this morning and they appear to working correctly again. I will send my job logs over to you this morning to verify and check.

[ReneSchumacher]

Hi all,

if I'm correctly informed, the product group has rolled back the deployment that caused the issue and the task should work again. Please let us know if you still see issues with BQC.

Thanks again for your patience and excellent collaboration! René

[aaroncearleytcms]

I have ran the build pipeline and it seems to be running well for me. thanks for the help @ReneSchumacher, @nishantms and @jjardina. let me know if you find some issues I might have missed.

[ReneSchumacher]

Closing this issue as it has been resolved and now additional issues have been reported.