Task Line coverage not matching open cover report

[d1820]

Seems the LIne and Branch Coverages do not match what is output in the coverage summary so to Check Code Coverage can fail, when the open cover results are valid

Summary of coverage

Logs

Configuration

Output when ran code just ran from powershell directly

There seems to be differences between them all, and this extension is lower then even the summary.

Commands Versions of code coverage ran

 # run Open Cover for code coverage
    & $nugetPackagesDir\opencover\4.6.519\tools\opencover.console.exe -target:"dotnet.exe" -targetargs:"test $($testDir.Name) -l:trx -r:`"..\TestResults`"" -output:$coverageFile -oldstyle -register:user "-filter:+[*]* -[*Infrastructure*]* -[*]*.Domain.* -[*Moq*]* -[*Microsoft.WindowsAzure.Storage*]* -[*]*.Dtos.* -[*Tests.Unit*]*" -searchdirs:$searchDirs -hideskipped -mergebyhash -mergeoutput "-excludebyattribute:*ExcludeFromCodeCoverage*"

# run ReportGenerator to create code coverage html
    & $nugetPackagesDir\reportgenerator\4.0.4\tools\net47\ReportGenerator.exe "-reports:$coverageFile" "-targetdir:coverage"

    # run OpenCoverToCoberturaConverter for use on build server
    & $nugetPackagesDir\opencovertocoberturaconverter\0.3.4\tools\OpenCoverToCoberturaConverter.exe -input:$coverageFile -output:cobertura.xml -includegetterssetters:true

Any ideas?

[ReneSchumacher]

Hey,

that looks pretty strange, esp. since all tools show different coverage values. In order to find the root cause, I'll need a couple more details. Could you please send me the coverage report generated by OpenCover? I'll then try to reproduce the issue in our environment. Send the information to our support email address PsGerExtSupport@microsoft.com.

If you only run once set of tests and generate only one cocverage file, you could try unchecking the "Force Filter" option to make sure that all coverage data is fully aggregated. Using "Force Filter" with an empty configuration and platform ignores coverage data that might be assigned to a specific config/platform.

Apart from that, I can only assume that the Cobertura converter generates different values, but I should be able to see that in my environment once I have your coverage file.

Thanks, René

[d1820]

Email Sent, In the email the results also included the removal of the "Force Filter"

[Blackbaud-ShaydeNofziger]

@d1820 did you ever find a resolution for this? We're experiencing similar issues where the line coverage does not match that in the Cobertura report.

[ReneSchumacher]

Hey,

so far, we haven't found the reason for this. The Build Quality Checks task does not calculate coverage values itself, it merely uses the values that have been published to Azure DevOps. If the Cobertura report isn't changed after it has been published, I can only assume that there is an issue with coverage data processing in the backend. Could you send us the Cobertura coverage in which you see the difference? Maybe we can reproduce the issue in our test environment and work with the testing tools team to resolve the issue.

Thanks, René

[d1820]

As a side note, i ended up just added a custom powershell task to the build

so we run the coverage, then look at the results and parse from there. I would prefer to use the tools here, but for the sake of time this was implemented

$b = "$env:CodeCoverageType"
$str = "(Visited\s$b).*?\((\d*[.\d]*)\)"
$regex = [regex] "$str"
$percent = 100

$matches = $regex.Matches($env:codeCoverageOutput)
if($matches.Count -gt 0 -and $matches[0].Groups.Count -gt 2){
    $percent = $($matches[0].Groups[2].Value)
}
if([decimal]$percent -lt $env:CodeCoverageThreshold){
    Write-Error -Message "Code coverage percentage below required threshold ($env:CodeCoverageThreshold). Current Percent: $($percent)"
}

[ReneSchumacher]

Our investigations have shown that this is an issue with OpenCover and not related to the Build Quality Checks task. We couldn't find the exact root cause but the issue seems to be related to coverage of async methods. If you encounter this issue, please contact the OpenCover support.