Build Quality Check for Code Coverage runs into a timeout

[soutchilin]

Hi,

Because I have not received any feedback on my post in an existing issue (https://github.com/MicrosoftPremier/VstsExtensions/issues/51) and I'm not sure if it got visible , please have this as a separate one.

Currently I have the "Waiting for code coverage data..." lasting till timeout, for Azure hosted pipeline, VSTest@2 and BuildQualityChecks@7. The code coverage file is created. Please see this output from VSTest@2 task. Results File: d:\a_temp\TestResults\VssAdministrator_fv-az674_2020-06-19_20_20_09.trx Attachments: d:\a_temp\TestResults\aa5e765f-4bc1-49ec-9fb4-f5809564f8f1\VssAdministrator_fv-az674 2020-06-19 20_19_59.coverage Test Run Successful. Total tests: 10 Passed: 10 Total time: 13.4534 Seconds Vstest.console.exe exited with code 0. **** Completed test execution ***** Test results files: d:\a_temp\TestResults\VssAdministrator_fv-az674_2020-06-19_20_20_09.trx Created test run: 1003014 Publishing test results: 10 Publishing test results to test run '1003014'. TestResults To Publish 10, Test run id:1003014 Test results publishing 10, remaining: 0. Test run id: 1003014 Published test results: 10 Publishing Attachments: 2 Completed TestExecution Model...

The next task is the BuildQuality one, that writes SystemVssConnection exists true Using IdentifierJobResolver Validating code coverage policy... Waiting for code coverage data...

and finally runs into a timeout. Waiting for code coverage data... [WARNING] Unable to get code coverage data within the maximum wait time.

[warning]Unable to get code coverage data within the maximum wait time.

Total lines: 0 Covered lines: 0 Code Coverage (%): 0 [ERROR] The code coverage value (0%, 0 lines) is lower than the minimum value (80%)!

[error]The code coverage value (0%, 0 lines) is lower than the minimum value (80%)!

Finishing: Code Coverage Treshold Check

Regards, Alexander

[ReneSchumacher]

Hi Alexander,

sorry that I missed your post in #51. There might be several reasons why the task runs into a timeout and a couple of those cannot be affected by us or our task. Sometimes, the backend in Azure DevOps is slow when merging code coverage values and doesn't finish before the task runs into a timeout. In that case, you may increase the timeout by setting the variable PSGer.Cover.MaxWaitTime to a value higher than 600000, which is the current default of ten minutes.

Note that you should only do so if you see coverage values in your build summary either directly on the summary page like this (e.g., for regular .NET applications)

or as a ReportGenerator report in the Code Coverage tab of the build summary like this (for non-.NET applications):

If you don't see coverage data in either place, there is something wrong with the coverage data generation or the publishing process.

René

[soutchilin]

Hi René,

Hmm, unfortunately I cannot recognized any tangible hint for me to do more analysis. The pipeline NEVER executed properly regarding the CC. The rest runs quick (2 minutes of [Build + Unit Tests]). Would it help for you to have a quick look on it, if I execute the pipeline with debug option activated?

For a CI pipeline it is no option to wait long for analysis. Sorry to get it straight: Do you mean, that the background check service usually takes some minutes until the pipeline get processed?

Regards, Alexander

[ReneSchumacher]

Hey,

the backend service merging code coverage data should usually only take a couple seconds depending on the amount of coverage data generated by your build. So in the very most cases, coverage data should be directly available when you run the Build Quality Checks task. We use the task very heavily internally as well and it usually just runs for a couple seconds and rarely has to wait for any background processing.

That is why I asked you to check if you can see coverage data in the build summary as shown by my screenshots. If the build doesn't display coverage, the data either hasn't be created correctly or something went wrong with the upload of the data to Azure DevOps.

I might be able to help if you send me your pipeline definition (esp. the configuration of your test task that creates the coverage data) and the logs of your pipeline preferably from a run for which the variable system.debug was set to true. You can either post the data here or send it via email to PSGerExtSupport@microsoft.com

René

[soutchilin]

Hi,

I see. I have another issue open, that the coverage file for C# doesn't get published. Maybe it has the same the root cause. Let me come back to you after that issue is solved.

Alexander

[ReneSchumacher]

Hi @soutchilin,

do you have an update on this issue? Has it been resolved or do you need additional help?

René

[ReneSchumacher]

Since we haven't received any feedback for this issue, I'm going to close it now. If your problem still persists, please feel free to post additional comments or create a new issue.

René

[slee2525FC]

Hi @ReneSchumacher . I am running into the same problem listed here. I am able to see the code coverage on my build pipeline but when I added the build task to add the code coverage policy, it just keeps telling me it's waiting for the code coverage report.

This is what the two build tasks look like

    - task: DotNetCoreCLI@2
        displayName: 'Test Solution'
        inputs: 
          command: test
          projects: '**/*Test/*.csproj'
          arguments: '--configuration $(BuildConfiguration) --settings $(Build.SourcesDirectory)\Test.runsettings --no-restore --no-build --collect "Code Coverage"'

      - task: BuildQualityChecks@7
        displayName: 'Verify code coverage'
        inputs:
          checkCoverage: true
          coverageFailOption: fixed
          coverageThreshold: '100'
          buildPlatform: $(BuildPlatform)

Please let me know if there is more information you might need.

Edit: Here are the results I get

[WARNING] read ECONNRESET
##[warning]read ECONNRESET
Total blocks: 0
Covered blocks: 0
Code Coverage (%): 0
[ERROR] The code coverage value (0%, 0 blocks) is lower than the minimum value (100%)!
##[error]The code coverage value (0%, 0 blocks) is lower than the minimum value (100%)!

[ReneSchumacher]

Hi @slee2525FC,

could you tell me in which environment you are using the task? Is this Azure DevOps Services (i.e., cloud) or on-premises Azure DevOps Server? Are you running your pipeline on a hosted or private agent? And if it's a private agent, are there any restrictions for connecting to the internet (e.g., proxy servers)?

The warning read ECONNRESET looks like there is a general communication issue. Perhaps you can run your pipeline again with the variables System.Debug and BQC.LogRawData set to true and send the task log files to PSGerExtSupport@microsoft.com? Perhaps they have more information about the root cause of that communication issue.

[pi3k14]

Same problem here with timeout (I think ECONNRESET was an temporary fault in the Azure infrastructure). Debug log showing upload of test run and coverage

2020-12-14T08:53:11.0841233Z ##[debug]File uploaded successsfully on LogStore X_2020-12-14_09_52_34.trx
2020-12-14T08:53:11.0842878Z ##[debug]File uploaded successsfully on LogStore X_2020-12-14.09_52_31.coverage

No hint in log from Quality check task about how the coverage data are accessed but 2020-12-14T08:53:13.4094596Z ##[debug]{"coverageData":[],"build":{"id":"44299","url":"https://dev.azure.com/XXX/_apis/build/Builds/44299"},"deltaBuild":null} seems correct.

Is there some mixups of the old and new url of Azure DevOps, xxx.visualstudio.com vs. dev.azure.com/xxx as the log also show 2020-12-14T08:53:12.7301251Z ##[debug]System.CollectionUri=https://xxx.visualstudio.com/

[ReneSchumacher]

Hi @pi3k14,

thanks for reporting this. The mix between old and new URLs is no problem. I guess that you are still using the old URL to access Azure DevOps (i.e., you didn't reconfigure your organization and switch to the new URL). However, both URLs work simultaneously, regardless of how you configured the organization. That's why all API calls to dev.azure.com/{org} work as well.

Could you please send an email to PSGerExtSupport@microsoft.com with the names of your organizations affected by the issue? Afaik, we still have some issues with the coverage merge job in Azure DevOps Services and might need to switch off a specific feature flag to mitigate the issue for your orgs. @slee2525FC Please also send us your organization name(s) via email so we can check if you are affected by the issue of the coverage merge job.

As long as you are affected, I'm going to reopen the issue again.

[slee2525FC]

Hi @ReneSchumacher,

Apologies for the delayed response. I am using Azure DevOps services on a hosted environment. I will email the name of my organization. I was able to get my code coverage report to work, but I had to change the output format to cobertura and add the explicit PublishCodeCoverageResults task to my pipeline.

From my understanding, this should not need be required to calculate the code coverage.

[pi3k14]

It is my impression that (almost?) all have bailed out to get this to work and instead are using coverlet. Getting to the bottom of the problem would be nice :)

[ReneSchumacher]

I haven't kept track of how many people moved to coverlet, but I guess that the future is going to be coverlet as it's fully cross-platform. Yet, it doesn't support all scenarios, afaik, and it shouldn't be necessary to use coverlet. From my discussions with the testing tools team I understood that they are using different storage mechanisms for coverage data depending on where the data comes from. And one of those mechanisms leads to a long delay in the processing of the data before BQC can read it again.

I'm starting to think about parsing the coverage data in the task to become independent from the Azure DevOps backend, which would also lead to the possibility to move BQC to GitHub Actions. Probably going to brood on that over the holidays.

[slee2525FC]

If it moves to GitHub Actions, would users that are using Azure DevOps still be able to take advantage of it?

[ReneSchumacher]

Yes, of course. As long as Azure Pipelines exist we will support it. My idea is to share as much code as possible between the two versions.

[pi3k14]

we still have some issues with the coverage merge job in Azure DevOps Services and might need to switch off a specific feature flag to mitigate the issue for your orgs @ReneSchumacher we got an e-mail stating that this feature flag was switched off. Testing showed no different behavior.

[ReneSchumacher]

Hi @pi3k14,

sorry I missed your comment! If the issue still persists, could you again send me the log files of a failing build that ran with the variables System.Debug and BQC.LogRawData set to true? Perhaps there's something else going wrong.

[pi3k14]

@ReneSchumacher - e-mail sent (finally).

As a side note. The build overview presents the coverage %, but to see the details you have to manually download the file and open it in Visual Studio Enterprise.

[ReneSchumacher]

I'm closing this issue again because I believe it's been resolved by the product group. If you still see timeouts, please let us know in a new issue or via email to PSGerExtSupport@microsoft.com,