search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.49k stars 2.61k forks source link

[Question]: Code coverage not working as expected #19597

Open rohanwayachal opened 7 months ago

rohanwayachal commented 7 months ago

Task name

VSTest

Task version

2

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

Azure DevOps Server (Please specify exact version in the textbox below)

Azure DevOps Server Version (if applicable)

No response

Operation system

windows

Question

after running multitarget build and test, code coverage for net462 is not getting published.

steps:
 - task: MSBuild@1
   displayName: 'Build solution'
   inputs:
    solution: src/Async.Bridges.sln
    msbuildArgs: '/p:TreatWarningsAsErrors="true" /restore'
    platform: 'Any CPU'
    configuration: '$(BuildConfiguration)' 

  - task: VSTest@3
    displayName: "Run Tests - .NET 6"
    inputs:
      testRunTitle: "Tests - .NET 6"
      searchFolder: '$(System.DefaultWorkingDirectory)\out\$(BuildConfiguration)'
      testAssemblyVer2: |
        **\net6.0\Tests\Async.Bridges.*Tests.dll
      vsTestVersion: 17.0
      runSettingsFile: src/Default.runsettings
      runInParallel: false
      runTestsInIsolation: true
      codeCoverageEnabled: true
      platform: "$(BuildPlatform)"
      configuration: "$(BuildConfiguration)"
      diagnosticsEnabled: true
      rerunFailedTests: true
      rerunFailedThreshold: 20
      rerunMaxAttempts: 2
      minimumExpectedTests: 1
      failOnMinTestsNotRun: true

  - task: VSTest@3
    displayName: "Run Tests - .NET Framework 4.6.2"
    inputs:
      testRunTitle: "Tests - .NET Framework 4.6.2"
      searchFolder: '$(System.DefaultWorkingDirectory)\out\$(BuildConfiguration)'
      testAssemblyVer2: |
        **\net462\Tests\Async.Bridges.*Tests.dll
      vsTestVersion: 17.0
      runSettingsFile: src/Default.runsettings
      runInParallel: false
      runTestsInIsolation: true
      codeCoverageEnabled: true
      platform: "$(BuildPlatform)"
      configuration: "$(BuildConfiguration)"
      diagnosticsEnabled: true
      rerunFailedTests: true
      rerunFailedThreshold: 20
      rerunMaxAttempts: 2
      minimumExpectedTests: 1
      failOnMinTestsNotRun: true

in ado ui, only coverage from net6.0 dll is published
rohanwayachal commented 7 months ago

image

joshua4163 commented 7 months ago

It appears that after running a multitarget build and test, the code coverage for .NET Framework 4.6.2 (net462) is not being published. Let's troubleshoot this issue.

  1. Check Output Folder: Ensure that the output folder for your test results and code coverage reports is correctly specified. In your VSTest tasks, you're using $(System.DefaultWorkingDirectory)\out\$(BuildConfiguration) as the search folder. Confirm that the test results and coverage files are indeed generated in this location.

  2. Verify Test Assembly Path: In the VSTest tasks, you've specified the test assembly pattern for both .NET 6 and .NET Framework 4.6.2. Make sure that the wildcard pattern **\net462\Tests\Async.Bridges.*Tests.dll matches the actual location of your test assemblies. Double-check the path and ensure that the test DLLs are present in the expected directory.

  3. Code Coverage Settings: Ensure that your runSettingsFile (located at src/Default.runsettings) includes the necessary configuration for code coverage. Specifically, check if the <DataCollector> section includes the CodeCoverage data collector. If not, add it to the run settings file.

  4. Permissions and Access: Verify that the build agent or self-hosted environment has the necessary permissions to read/write files in the specified output folder. Sometimes issues arise due to insufficient permissions.

  5. Logging and Diagnostics: Enable detailed logging and diagnostics for the VSTest tasks. Set diagnosticsEnabled: true to get more information about what's happening during the test execution. Review the logs for any relevant error messages or warnings related to code coverage.

  6. Rerun Failed Tests: You've configured the tasks to rerun failed tests (rerunFailedTests: true). If any tests fail during the initial run, they will be retried. Check if any tests are consistently failing and investigate why.

  7. Thresholds and Minimum Expected Tests: You've set thresholds for rerunning failed tests (rerunFailedThreshold) and specified a minimum expected test count (minimumExpectedTests). Ensure that these values align with your expectations. If the minimum expected tests are not met, the build may fail.

  8. Missing Value for rerunFailedThreshold: It seems there's a missing value for rerunFailedThreshold in your configuration. Please provide the appropriate value (e.g., 20 as shown in your .NET 6 task).