Truncated or corrupted coverage data leading to many broken test runs

[AArnott]

Description

I get a surprisingly high number of test runs that fail because of this error, after an otherwise successful test run:

Calculating coverage result...
D:\a\_work\_temp\.nuget\packages\coverlet.msbuild\3.1.2\build\coverlet.msbuild.targets(71,5): error : Unable to read beyond the end of the stream. [D:\a\_work\1\s\test\Microsoft.VisualStudio.Composition.Analyzers.Tests\Microsoft.VisualStudio.Composition.Analyzers.Tests.csproj]
D:\a\_work\_temp\.nuget\packages\coverlet.msbuild\3.1.2\build\coverlet.msbuild.targets(71,5): error :    at System.IO.BinaryReader.InternalRead(Int32 numBytes) [D:\a\_work\1\s\test\Microsoft.VisualStudio.Composition.Analyzers.Tests\Microsoft.VisualStudio.Composition.Analyzers.Tests.csproj]
D:\a\_work\_temp\.nuget\packages\coverlet.msbuild\3.1.2\build\coverlet.msbuild.targets(71,5): error :    at Coverlet.Core.Coverage.CalculateCoverage() in /_/src/coverlet.core/Coverage.cs:line 414 [D:\a\_work\1\s\test\Microsoft.VisualStudio.Composition.Analyzers.Tests\Microsoft.VisualStudio.Composition.Analyzers.Tests.csproj]
D:\a\_work\_temp\.nuget\packages\coverlet.msbuild\3.1.2\build\coverlet.msbuild.targets(71,5): error :    at Coverlet.Core.Coverage.GetCoverageResult() in /_/src/coverlet.core/Coverage.cs:line 161 [D:\a\_work\1\s\test\Microsoft.VisualStudio.Composition.Analyzers.Tests\Microsoft.VisualStudio.Composition.Analyzers.Tests.csproj]
D:\a\_work\_temp\.nuget\packages\coverlet.msbuild\3.1.2\build\coverlet.msbuild.targets(71,5): error :    at Coverlet.MSbuild.Tasks.CoverageResultTask.Execute() in /_/src/coverlet.msbuild.tasks/CoverageResultTask.cs:line 83 [D:\a\_work\1\s\test\Microsoft.VisualStudio.Composition.Analyzers.Tests\Microsoft.VisualStudio.Composition.Analyzers.Tests.csproj]

Any idea why? Could it be that the process running the tests terminates before coverage data has been flushed to disk?

Diagnostic logs

testResults-Windows.zip

Environment

Azure Pipelines

[MarcoRossignoli]

It's a known issue of coverlet that relies on process exits to save probes. You should move to the datacollector integration https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#vstest-stops-process-execution-early

You can also use: dotnet test --collect "Code Coverage;Format=cobertura" or the dotnet tool https://docs.microsoft.com/en-us/dotnet/core/additional-tools/dotnet-coverage they're not suffering the issue because they rely on shared memory. They're supported on Windows(x86, x64 and arm64), Linux(x64) and macOS(x64) and they're using dynamic instrumentation(no assemblies rewrite).

[MarcoRossignoli]

We added also a workaround a pair of months ago https://github.com/microsoft/vstest/blob/main/src/Microsoft.TestPlatform.CrossPlatEngine/Client/ProxyOperationManager.cs#L311 where you can setup the VSTEST_TESTHOST_SHUTDOWN_TIMEOUT environment variable to extend the defaul 100ms timeout. It should be in 17.2 https://github.com/Microsoft/vstest-docs/blob/main/docs/releases.md#v1720-preview-20220401-07 cc: @Evangelink

[AArnott]

Thanks, @MarcoRossignoli. Your first link ultimately leads me to https://github.com/coverlet-coverage/coverlet#vstest-integration-preferred-due-to-known-issue-supports-only-net-core-application, which says it only works on .NET Core. I need to test and collect code coverage for .NET Framework too. So should I use the options in your second paragraph instead? I'm thinking adding --collect "Code Coverage;Format=cobertura" to my existing dotnet test command will be super simple.

Are there any other changes I should make? My test projects typically contain:

    <PackageReference Include="coverlet.msbuild" Version="3.1.2" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.2.0" />
</ItemGroup>
<PropertyGroup>
    <CoverletOutputFormat>cobertura</CoverletOutputFormat>
    <Exclude>[xunit.*]*</Exclude>
    <!-- Ensure we preserve each coverlet output file per target framework: https://github.com/tonerdo/coverlet/issues/177 -->
    <CoverletOutput>$(OutputPath)/</CoverletOutput>

And I spawn dotnet test with the -p:CollectCoverage=true switch. Should I continue to do that?

[MarcoRossignoli]

which says it only works on .NET Core. I need to test and collect code coverage for .NET Framework too.

Sorry it's not up to date...it's working also with .NET Framework(we moved collector to netstandard2.0), just updated.

The setup for collector is here https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md you can remove coverlet.msbuild that is used only to inject the MSBuild Tasks.

And I spawn dotnet test with the -p:CollectCoverage=true switch. Should I continue to do that?

Nope with collector you should use runsettings https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md#coverlet-options-supported-by-vstest-integration and run with dotnet test --collect:"XPlat Code Coverage"

Sample https://github.com/coverlet-coverage/coverlet/tree/master/Documentation/Examples/VSTest/HelloWorld

I'm thinking adding --collect "Code Coverage;Format=cobertura" to my existing dotnet test command will be super simple.

If you move to dotnet-coverage the settings are described here https://docs.microsoft.com/en-us/dotnet/core/additional-tools/dotnet-coverage#settings

[AArnott]

dotnet-coverage merge is buggy. It drops covered lines that are in at least some of the input reports. Where should I file that bug?

[AArnott]

I can't find the repo with the dotnet-coverage tool source code in it, but I see plenty of issues with that tool filed in this repo, so I filed https://github.com/microsoft/vstest/issues/3732.

[MarcoRossignoli]

I'm going to close this one because the correct issue for the merge is already opened and the corrupted issue was been fixed by build-in code coverage tool.