Code coverage seems to be broken in version 17.10.0

norbert-groen commented 1 month ago

Description

In our CI pipeline in Azure Devops (on premise), we use the latest stable Visual Studio test platform (currently 17.10.0): Although CodeCoverage states:

CodeCoverage.exe is deprecated. Use dotnet-coverage global tool or Microsoft.CodeCoverage.Console instead. Some features of CodeCoverage.exe will be removed in next release of Visual Studio.

Azure Devops seems to use this tool. The code coverage is broken and does not represent proper coverage anymore. Another tool, that is using this tool, is Sonarqube (our code quality gateway). These errors appears:

##[error]Failed to convert the binary code coverage reports to XML. No code coverage information will be uploaded to the server (SonarQube/SonarCloud).
##[error]Check that the downloaded code coverage file (D:\azagent-p\A1\_work\_temp\11e7728a-01e7-4ae5-a924-a480fa3e0087\<---->_2024-05-27.18_12_16.coverage) is valid by opening it in Visual Studio. If it is not, check that the internet security settings on the build machine allow files to be downloaded from the Team Foundation Server machine.

and the gateway fails. This results in impossibility to merge code via pull requests, while the gateway is mandatory. Visual studio can interpret the coverage, but it doesn't represent the proper coverage.

Steps to reproduce

Use the latest stable version in a CI pipeline

  displayName: 'Visual Studio Test Platform Installer'
   inputs:
     packageFeedSelector: 'nugetOrg'
     versionSelector: 'latestStable'

and compare the code coverage against the same pipeline, but run with configuration:

  displayName: 'Visual Studio Test Platform Installer'
   inputs:
    packageFeedSelector: 'nugetOrg'
    versionSelector: 'specificVersion'
    testPlatformVersion: '17.9.0'

Expected behavior

Good representation of code coverage. External tools should be able to interpret coverage results.

Actual behavior

Missing parts of coverage and external tools unable to interpret results.

Environment

Running agent on Windows: 10.0.17763 Current agent version: '3.225.2'

michha commented 1 month ago

We encountered the same problem. Our pipeline agents run inside container. Image from 15th of may (after patch tuesday) is fine. But with an image from 22th of may (after Build Event with VS 17.10.) we got the error message described above. Reverting to the previous image restored code coverage functionality.

norbert-groen commented 1 month ago

Sonar is aware of codecoverage.exe deprecating and are migrating away of its using. But is Microsoft too? How to fix the code coverage in the CI pipeline? We are using: dotnet test --collect:"Code Coverage" to collect and this is broken.

fhnaseer commented 1 month ago

Can you please use this in AzDO and collect code coverage? Collect code coverage inside Azure DevOps Pipelines

almolo1 commented 1 month ago

I'm also seeing this error. All I'm trying to do is to run SonarCloudAnalyze in Azure DevOps pipeline like this:

 - task: SonarCloudAnalyze@2
    displayName: Sonar Analysis

I’m assuming that behind the scenes this uses the latest available version (17.10.0) But since I’m not using VisualStudioTestPlatformInstaller task or anything like that, I don’t know how to rollback from 17.10.0 to 17.9.0… Any suggestions?

Note that originally I used SonarCloudAnalyze@1 and that generated pipeline output like this:

Executing file C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe
  Args: analyze /output:C:\a\3\s\TestResults\AzDevOps_aisar90cd000BJK_2024-06-05_13_02_40\In\aisar90cd000BJK\AzDevOps_aisar90cd000BJK_2024-06-05.13_02_41.cobertura.coveragexml C:\a\3\s\TestResults\AzDevOps_aisar90cd000BJK_2024-06-05_13_02_40\In\aisar90cd000BJK\AzDevOps_aisar90cd000BJK_2024-06-05.13_02_41.cobertura.xml 
  Working directory: C:\a\3\s\TestResults\AzDevOps_aisar90cd000BJK_2024-06-05_13_02_40\In\aisar90cd000BJK
  Timeout (ms):60000
  Process id: 5500
CodeCoverage.exe is deprecated. Use dotnet-coverage global tool or Microsoft.CodeCoverage.Console instead. Some features of CodeCoverage.exe will be removed in next release of Visual Studio.

Process returned exit code -1
Coverage report conversion completed successfully.
##[error]Failed to convert the binary code coverage reports to XML. No code coverage information will be uploaded to the server (SonarQube/SonarCloud).
Check that the downloaded code coverage file (C:\a\3\s\TestResults\AzDevOps_aisar90cd000BJK_2024-06-05_13_02_40\In\aisar90cd000BJK\AzDevOps_aisar90cd000BJK_2024-06-05.13_02_41.cobertura.xml) is valid by opening it in Visual Studio. If it is not, check that the internet security settings on the build machine allow files to be downloaded from the Team Foundation Server machine.
Failed to convert the binary code coverage reports to XML. No code coverage information will be uploaded to the server (SonarQube/SonarCloud).

Then I switched to v2, which looks a little bit better, but ultimately still fails to upload code coverage to SonarCloud

Converting coverage file 'C:\a\14\s\TestResults\AzDevOps_aisar90cd000BJ6_2024-06-05_13_35_33\In\aisar90cd000BJ6\AzDevOps_aisar90cd000BJ6_2024-06-05.13_35_33.cobertura.xml' to 'C:\a\14\s\TestResults\AzDevOps_aisar90cd000BJ6_2024-06-05_13_35_33\In\aisar90cd000BJ6\AzDevOps_aisar90cd000BJ6_2024-06-05.13_35_33.cobertura.coveragexml'.
Coverage report conversion completed successfully.
##[error]Failed to convert the binary code coverage reports to XML. No code coverage information will be uploaded to the server (SonarQube/SonarCloud).
Check that the downloaded code coverage file (C:\a\14\s\TestResults\AzDevOps_aisar90cd000BJ6_2024-06-05_13_35_33\In\aisar90cd000BJ6\AzDevOps_aisar90cd000BJ6_2024-06-05.13_35_33.cobertura.xml) is valid by opening it in Visual Studio. If it is not, check that the internet security settings on the build machine allow files to be downloaded from the Team Foundation Server machine.
Failed to convert the binary code coverage reports to XML. No code coverage information will be uploaded to the server (SonarQube/SonarCloud).

As you can see it no longer complains about CodeCoverage.exe being deprecated and there is no process exit code -1. There’s even a line saying “Coverage report conversion completed successfully.” But then it fails again.

Any suggestions how to get it running in Azure DevOps pipeline like this?

fhnaseer commented 1 month ago

There are two things here.

Collect code coverage => It seems like SonarCloud was using CodeCoverage.exe in SonarCloudAnalyze@1, and then using something else in SonarCloudAnalyze@2. Hence there is no error in @2. CodeCoverage.exe is deprecated and alternatives should be used to collect code coverage as mentioned in the error message.
Convert coverage report => There are many possibilities here 2.1. CodeCoverage.exe /analyze => Since CodeCoverage.exe is deprecated and SonarCloud is using this to convert then it might not work. 2.2. Some other APIs => It depends what SonarCloud is using to convert coverage reports. Depending on the tool, it could be broken. 2.3. dotnet-coverage tool can be used to to convert coverage files as well. dotnet-coverage merge input.coverage --output-format cobertura --output converted.cobertura.xml

I would suggest you to create an issue in SonarCloud repo. They can answer what they are using and which component is failing in your scenario.

norbert-groen commented 1 month ago

The new version (6) of the Sonar extension does not use the CodeCoverage anymore. So this issue can be closed.

microsoft / codecoverage