Check build quality

[SuryaPractise]

Hi Team, Check build quality failing with below errors,Can you help me with solution.

[section]Starting: Check build quality

============================================================================== Task : Build Quality Checks Description : Breaks a build based on quality metrics like number of warnings or code coverage. Version : 5.2.4 Author : Microsoft Premier Services Help : [Docs]

SystemVssConnection exists true Validating code coverage policy... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data...

-

Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Unable to get code coverage data within the maximum wait time. Total blocks: 0 Covered blocks: 0 Code Coverage (%): 0

[error]At least one build quality policy was violated. See Build Quality Checks section for more details.

[section]Finishing: Check build quality

[SuryaPractise]

enabled system.debug & BQC.LogRawData,Below is few lines from logs.

Till date it was working fine from couple hours it started failing.

Waiting for code coverage data...

[debug]-------------------- RAW DATA --------------------

[debug]buildCoverageSummary:

[debug]{"coverageData":[],"build":{"id":"31882","url":"https://-------------/_apis/build/Builds/31882"},"deltaBuild":null}

[debug]------------------ END RAW DATA ------------------

Waiting for code coverage data...

[debug]-------------------- RAW DATA --------------------

[debug]buildCoverageSummary:

[debug]{"coverageData":[],"build":{"id":"31882","url":"https://t----------/_apis/build/Builds/31882"},"deltaBuild":null}

[debug]------------------ END RAW DATA ------------------

[ReneSchumacher]

Hi @SuryaPractise,

thanks for the log snippets. Apparently, the task doesn't find the coverage data anymore. Did you change anything in your build definition? Are you running in the cloud or on premises? If this is a cloud setup, could you try to update to the latest version of the task (6.1.2) and run your build again? There have been changes in the backend that might result in problems with the older versions of the task. If you're running on-prem, please let me know what version of Team Foundation Server or Azure DevOps Server you're running.

In addition, can you make sure that you still see coverage data in your build result?

Thanks, René

[SDanehy]

We just started seeing these issues too. It started a couple hours ago. I updated to the latest version, 6.1.2, and it us still not working. I am no longer seeing code coverage in my build results. I am seeing all tests pass just not the coverage

Thank you, Shane

[ReneSchumacher]

Hi @SDanehy,

thank you for letting us know that the issue seems to be spreading. In this case, I can only assume that the root cause is somewhere in the Azure DevOps Services backend and not in the task itself. To test my assumption could you please open the following URI for one of your already finished builds (just replace organization,m project and buildId) and check if there really is no code coverage reported: https://dev.azure.com/{organization}/{project}/_apis/test/codecoverage?buildId={buildId}&api-version=5.1-preview.1

I'll forward the issue to the Azure DevOps testing tools team.

Thank you, René

[SuryaPractise]

Hi Rene, please find my answers in line.

1. We are in cloud,everything hosted on Azure.

2. Build Quality Checks Extension installed in our setup is 6.12.

3. Tried (just replace organization,m project and buildId) and check if there really is no code coverage reported: https://dev.azure.com/{organization}/{project}/_apis/test/codecoverage?buildId={buildId}&api-version=5.1-preview.1 by replacing organization,m project and buildId,getting 404 page not found error.

Thanks, Surya

[SuryaPractise]

Please find the output from last successful CI build

{"coverageData":[{"coverageStats":[{"label":"Blocks","position":1,"total":44992,"covered":34154,"isDeltaAvailable":true,"delta":0.01},{"label":"Lines","position":2,"total":18412,"covered":13425,"isDeltaAvailable":true,"delta":0.0}],"buildPlatform":"","buildFlavor":""}],"build":{"id":"31854","url":"https://dev.azure.com/---------/_apis/build/Builds/31854"},"deltaBuild":null,"status":"finalized"}

[SuryaPractise]

Please find response using the URI for CI job which is failing.

{"coverageData":[],"build":{"id":"31882","url":"https://dev.azure.com/----------/_apis/build/Builds/31882"},"deltaBuild":null}

[ReneSchumacher]

Hi @SuryaPractise,

your data confirms my assumption that the coverage data is missing or not correctly associated with the build. Since our task does neither calculate the coverage nor parse the coverage files itself but only reads the values from the Azure DevOps backend, we cannot fix the issue. I already informed the appropriate team and hope they can fix the issue quickly. Until then, please disable the task or check its "continue on error" control option to make your builds pass again.

René

[SuryaPractise]

Hi Rene,

Can you share ETA for this? Its blocking our code quality check.

For now we can disable task and run pipeline.

Thanks, Surya

[ReneSchumacher]

I haven't received any feedback from the team yet and they probably won't investigate over the weekend. To speed things up I opened a support request in the developer community forms as well (see https://developercommunity.visualstudio.com/content/problem/784837/code-coverage-data-missing-from-build.html). Please add more detail to the thread if you can share more information with us (e.g., when did you see the issue for the first time etc.).

I'll update the issue as soon as I have something to share.

[mleziva]

I was experiencing the same problem when using the task Visual Studio Test version 1. Changing to use version 2. resolved my issues

[ssanderlin]

I was experiencing the same problem when using the task Visual Studio Test version 1. Changing to use version 2. resolved my issues

+1 updating the VSTest task to v2 solved the issue for me as well. Thank you, @mleziva

[dwilsonbst]

Currently have this same issue. Using dotnet test with coverlet. Produces a code coverage report. Lines are identified fine, but there are 0 branches in code so 0/0 fails. Snippets of log from same build: Line Coverage: Validating code coverage policy... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Successfully read code coverage data from build. Total lines: 5 Covered lines: 4 Code Coverage (%): 80 Finishing: Check Code Coverage Line

Branch Coverage: Validating code coverage policy... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Waiting for code coverage data... Successfully read code coverage data from build. Total branches: 0 Covered branches: 0 Code Coverage (%): 0

[error]At least one build quality policy was violated. See Build Quality Checks section for more details.

Finishing: Check Code Coverage Branch

[ASH-Anthony]

Getting a similar issue as @dwilsonbst if there are no unit tests (we have UI tests in our suite as well).

...
Waiting for code coverage data...
Waiting for code coverage data...
##[error]The task has timed out.
Finishing: Check build quality

This is a UI only (JS) component so, there aren't any functions to test. When I add a dummy function and unit test, this works. We have a common task group for our components, which is why this is running with 0 unit tests

[ReneSchumacher]

@dwilsonbst Are you sure that coverlet is correctly creating branch coverage? You can check by opening the following URL in your browser: https://dev.azure.com/{organization}/{project]/_apis/test/codecoverage?buildId={buildId}

Simply replace organization, project, and buildId with your values (just use the ID of the build that reported zero branch coverage). The result should look like this:

{"coverageData":[{"coverageStats":[{"label":"Branches","position":6,"total":380,"covered":251,"isDeltaAvailable":true,"delta":0.18},{"label":"Lines","position":4,"total":912,"covered":700,"isDeltaAvailable":true,"delta":0.0}],"buildPlatform":"","buildFlavor":""}],"build":{"id":"1805","url":"https://dev.azure.com/myorg/_apis/build/Builds/1805"},"deltaBuild":null,"status":"finalized"}

If the result does not have a coverageStats object with the label branch or branches our task cannot get any branch value.

[ReneSchumacher]

@ASH-Anthony Whenever you put the BQC task into a build it assumes that there should be code coverage. Thus, if you use it in a build that doesn't create coverage data, it will always assume zero coverage. Unless you specify a minimum coverage threshold, though, this shouldn't be a problem. However, it might slow down your build.

According to your log it seems that the task has been canceled by the build system. The error message "The task has timed out" is not created by the task itself. When we run into a timeout we issue a warning. Did you manually set a timeout for this task using the control options?

[dwilsonbst]

@ReneSchumacher I think that is the problem I'm trying to convey. Code coverage was generated 100% ok. The code itself does not have any branches. However we want branch code coverage. The idea being that we want the build set up so that should anyone intrude a code branch then it the step would automatically be there and not require build changes to account for that.

Result of a build without a branch, but code coverage working fine for lines: {"coverageData":[{"coverageStats":[{"label":"Lines","position":4,"total":5,"covered":4,"isDeltaAvailable":false,"delta":0.0}],"buildPlatform":"","buildFlavor":""}],"build":{"id":"4969","url":"https://dev.azure.com/org/_apis/build/Builds/4969"},"deltaBuild":null,"status":"finalized"}

Same build configuration after a "fake" code coverage branch was added in src.

{"coverageData":[{"coverageStats":[{"label":"Branches","position":6,"total":2,"covered":2,"isDeltaAvailable":false,"delta":0.0},{"label":"Lines","position":4,"total":7,"covered":6,"isDeltaAvailable":false,"delta":0.0}],"buildPlatform":"","buildFlavor":""}],"build":{"id":"4973","url":"https://dev.azure.com/org/_apis/build/Builds/4973"},"deltaBuild":null,"status":"finalized"}

[ASH-Anthony]

@ReneSchumacher yes, we have a 1 minute timeout and there is a coverage report uploaded. The issue is when there are 0 lines to cover. If I add a dummy function (again, the component only renders HTML), it passes.

With error this is our Publish Code Coverage task:

Lines- 0 of 0 covered.
Branches- 0 of 0 covered.
Modifying Cobertura Index file

After adding dummy function

- Lines- 0 of 0 covered.
+ Lines- 1 of 1 covered.
Branches- 0 of 0 covered.
Modifying Cobertura Index file

Adding the dummy function and a unit test for it is the only difference between this failing and working

[ReneSchumacher]

@dwilsonbst Can you tell me which version of the BQC task you are using and how you configured it? I do understand your goal. Yet, I'm not sure if I understand your expectations.

Let's assume you add a BQC task to you pipeline, configure it to look for branch coverage, and set it up to fail based on the previous build. In other words, the task looks for branch coverage, doesn't find any, and simply assumes a value of zero. Since this will happen for the current build and the baseline build, the policy will pass with zero coverage.

If, however, you specify a minimum coverage threshold, the policy has to fail since there is no coverage. Imho, this is the best behavior as no coverage clearly is unexpected if you tell the task to expect a certain amount of coverage.

Is that the problem you are running into? If so, we could add an option to simply pass the policy if no coverage data can be found. I believe, though, that this might lead to some false positive policy results if there is an issue reading code coverage from the backend, which still happens from time to time.

[ReneSchumacher]

@ASH-Anthony I do understand your situation, but you are actually experiencing two different issues here.

First, there is an issue with the code coverage backend in Azure DevOps. A while ago the product teams changed the way coverage files are processed and added a new status property to the JSON response of our coverage API. That status property should show if there coverage data is present, currently being processed, or already aggregated and stable. Based on the presence of the status property our task uses one of two different mechanisms to wait for the code coverage to be fully processed in the backend. If coverage data and status is present and the status indicates stable coverage values, we use the data right away, which is what you're probably seeing after adding your dummy function and test.

However, if there is no coverage data (or the status property is missing (sometimes happens) or indicates that coverage data is being processed) our task keeps polling for coverage data until it reaches an internal timeout and gives up. This leads to your second issue because you are using a very short task timeout. Depending on the waiting mechanism used our task might wait for up to a minute between two polls for coverage data. And since it doesn't log any information during that idle time, it might get canceled by the agent.

Without additional log data I cannot really be sure what exactly happens in your pipeline but the issue should be resolved by slightly increasing the task timeout to perhaps 90 seconds.

[ASH-Anthony]

Without additional log data I cannot really be sure what exactly happens in your pipeline but the issue should be resolved by slightly increasing the task timeout to perhaps 90 seconds.

Thanks for your answer. Unfortunately, changing the timeout up to 4 minutes still did not work. We may just have to create a separate task group for UI components like this, unless there are exposed variables in the coverage report (e.g., gt([totalBranches], 0))

[dwilsonbst]

@ReneSchumacher, I'm using version 5 it seems.

I understand the problem that arises from just letting it pass when there are 0/0 of a coverage line. I guess what i'm looking for is more of a warning that there is no branch coverage vs an error failing the build. My thought process is the fact that I have code coverage (line), it should/could be able to realize that code coverage was in fact generated properly, there just isn't any branches to cover. I'd almost expect 0/0 to be 100% though I see the train of thought 0/0 being 0% as well.

Currently the Build Quality Checks task can only check one thing (line, branch, etc). Because of this it seems like it doesn't know if 0/0 is because coverage was uploaded bad or if there are just 0/0 branches. Maybe if a single step could check for both branch and line + any others, it might be able to make a more informed decision on failure/warning; ie coverage has lines but has 0/0 branches would be a warning about branches or even passing 100% rather than complete failure when everything is 0/0.

This is not a show stopper necessarily, but was hoping to get some insight and a fix/workaround if one could be made available. For now we've added an arbitrary branch to the code and it has resolved in its one case, we could also just allow failure on that step until branches are introduced later as well.

[ReneSchumacher]

@dwilsonbst Thanks for the feedback. In fact, the task can distinguish between cases where there's no coverage data at all and cases where coverage data has been read but there is no coverage of the requested type. In the first case you'd see a warning that coverage data could not be read within the timeout.

In both cases, though, we decided to let the policy continue the evaluation and just assume zero coverage as this generally should work. The only exception is using a minimum coverage threshold, which I guess you have configured, right? Otherwise, I don't understand why the policy is failing in your case.

If your issue really is tied to the minimum coverage threshold, we could perhaps add an option to treat 0/0 elements as 100% instead of 0%. Would that help? I think it's flexibel enough to let people decide how to treat this situation. I'm not sure if we should issue a warning if the requested coverage type is not available. @woehrl01 What do you think?

[dwilsonbst]

@ReneSchumacher, that is exactly the situation. We expect 80% line and 80% branch coverage. So the build fails on 0/0 = 0%. An option to treat 0/0 as 100% would be perfect for this use case I think.

[woehrl01]

@ReneSchumacher I agree treating 0/0 as 100% is a good fit in that case.

[ReneSchumacher]

Alright, will do. I just added this to our backlog. It might take a couple days for us to add this as we're also in the progress of adding arbitrary file warning analysis based on previous build. I'll let you know when the feature has been added.

[dwilsonbst]

Wonderful thanks!

[ReneSchumacher]

Hey @dwilsonbst,

we just finished work on the new version 6.5.0 that now has the requested option. I'll be doing some final testing and if all goes well the new version should be published in a couple minutes.

[dwilsonbst]

@ReneSchumacher, awesome! I look forward to having the option, Should the version require any update on our end (extension updates/install), or just automatic?

[ReneSchumacher]

It will require you to update the task version. We had to add a couple more changes and introduced one breaking change (we changed the default coverage delta type from absolute to percentage, because that is what most customers use). Thus, the new version will be 7.0.0. If you're using YAML, you need to change from BuildQualityChecks@6 to BuildQualityChecks@7. In the UI you need to update the pinned task version in the task UI.

[soutchilin]

Hello,

Currently I have the same "Waiting for code coverage data..." for Azure hosted pipeline, VSTest@2 and BuildQualityChecks@7. The code coverage file is created. Please see this output from VSTest@2 task. Results File: d:\a_temp\TestResults\VssAdministrator_fv-az674_2020-06-19_20_20_09.trx Attachments: d:\a_temp\TestResults\aa5e765f-4bc1-49ec-9fb4-f5809564f8f1\VssAdministrator_fv-az674 2020-06-19 20_19_59.coverage Test Run Successful. Total tests: 10 Passed: 10 Total time: 13.4534 Seconds Vstest.console.exe exited with code 0. **** Completed test execution ***** Test results files: d:\a_temp\TestResults\VssAdministrator_fv-az674_2020-06-19_20_20_09.trx Created test run: 1003014 Publishing test results: 10 Publishing test results to test run '1003014'. TestResults To Publish 10, Test run id:1003014 Test results publishing 10, remaining: 0. Test run id: 1003014 Published test results: 10 Publishing Attachments: 2 Completed TestExecution Model...

The next task is the BuildQuality one, that writes SystemVssConnection exists true Using IdentifierJobResolver Validating code coverage policy... Waiting for code coverage data...

and finally runs into a timeout. Waiting for code coverage data... [WARNING] Unable to get code coverage data within the maximum wait time.

[warning]Unable to get code coverage data within the maximum wait time.

Total lines: 0 Covered lines: 0 Code Coverage (%): 0 [ERROR] The code coverage value (0%, 0 lines) is lower than the minimum value (80%)!

[error]The code coverage value (0%, 0 lines) is lower than the minimum value (80%)!

Finishing: Code Coverage Treshold Check

Regards, Alexander

[ReneSchumacher]

I'm closing this issue for now. If you still see sporadic timeouts, please open a new issue so we can track them. Since the root cause for this issue lies in Azure DevOps Services and not in our task, we cannot really help with the issue. However, if there are enough occurrences of this problem, chances will be higher that the product team might fix the issue once and for all.