/p:CollectCoverage=true causes all test discovery to fail when class derives from XUnitVerifier

[AArnott]

Repro Unzip: mytestlib.zip Within the unzipped directory, run:

dotnet test
dotnet test /p:CollectCoverage=true

Notice how the one test method runs successfully the first time. But when code coverage is enabled, no tests are discovered at all.

I suspect an exception is being thrown during the code coverage instrumentation. Perhaps the XunitVerifier class I'm using as a base class (which is required to repro the failure) uses a pattern that coverlet.msbuild can't deal with.

[AArnott]

@sharwell owns the XunitVerifier class I believe.

[MarcoRossignoli]

Hi @AArnott, yep the issue is the instrumentation of Microsoft.CodeAnalysis.Testing.Verifiers.XUnit.dll For now you can fix with filters

dotnet test /p:CollectCoverage=true  /p:exclude=\"[Microsoft*]*,[xunit*]*\"
dotnet test /p:CollectCoverage=true  /p:include=\"[mytestlib*]*\"

BTW next release should solve without any action we'll skip non instrumentable dll(embedded or external pdb with no local source), I did a test with master(tomorrow you can use nightly, I just fixed an issue we're back by a week). Future result will be:

C:\git\coverletissue\mytestlib
λ dotnet test /p:CollectCoverage=true /p:CoverletToolsPath=C:\git\coverlet\src\coverlet.msbuild.tasks\bin\Debug\netstandard2.0\
C:\Users\Marco\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(24,5): warning : [coverlet] Unable to instrument module: C:\git\coverletissue\mytestlib\bin\Debug\net472\Microsoft.CodeAnalysis.Analyzer.Testing.dll, embedded pdb without local source files, [/_/src/Tools/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.Analyzer.Testing/AnalyzerTest`1.cs] [C:\git\coverletissue\mytestlib\mytestlib.csproj]
C:\Users\Marco\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(24,5): warning : [coverlet] Unable to instrument module: C:\git\coverletissue\mytestlib\bin\Debug\net472\Microsoft.CodeAnalysis.CodeFix.Testing.dll, embedded pdb without local source files, [/_/src/Tools/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.CodeFix.Testing/CodeFixTest`1.cs] [C:\git\coverletissue\mytestlib\mytestlib.csproj]
C:\Users\Marco\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(24,5): warning : [coverlet] Unable to instrument module: C:\git\coverletissue\mytestlib\bin\Debug\net472\Microsoft.CodeAnalysis.CSharp.CodeFix.Testing.dll, embedded pdb without local source files, [/_/src/Tools/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.CSharp.CodeFix.Testing/CSharpCodeFixTest`3.cs] [C:\git\coverletissue\mytestlib\mytestlib.csproj]
C:\Users\Marco\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(24,5): warning : [coverlet] Unable to instrument module: C:\git\coverletissue\mytestlib\bin\Debug\net472\Microsoft.CodeAnalysis.CSharp.CodeFix.Testing.XUnit.dll, embedded pdb without local source files, [/_/src/Tools/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.CSharp.CodeFix.Testing.XUnit/CodeFixVerifier.cs] [C:\git\coverletissue\mytestlib\mytestlib.csproj]
C:\Users\Marco\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(24,5): warning : [coverlet] Unable to instrument module: C:\git\coverletissue\mytestlib\bin\Debug\net472\Microsoft.CodeAnalysis.Testing.Verifiers.XUnit.dll, embedded pdb without local source files, [/_/src/Tools/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.Testing.Verifiers.XUnit/XUnitVerifier.cs] [C:\git\coverletissue\mytestlib\mytestlib.csproj]
C:\Users\Marco\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(24,5): warning : [coverlet] Unable to instrument module: C:\git\coverletissue\mytestlib\bin\Debug\net472\xunit.assert.dll, embedded pdb without local source files, [C:\Dev\xunit\xunit\src\xunit.assert\Asserts\Assert.cs] [C:\git\coverletissue\mytestlib\mytestlib.csproj]
C:\Users\Marco\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(24,5): warning : [coverlet] Unable to instrument module: C:\git\coverletissue\mytestlib\bin\Debug\net472\xunit.core.dll, embedded pdb without local source files, [C:\Dev\xunit\xunit\src\common\ExceptionExtensions.cs] [C:\git\coverletissue\mytestlib\mytestlib.csproj]
C:\Users\Marco\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(24,5): warning : [coverlet] Unable to instrument module: C:\git\coverletissue\mytestlib\bin\Debug\net472\xunit.execution.desktop.dll, embedded pdb without local source files, [C:\Dev\xunit\xunit\src\xunit.execution\Extensions\ReflectionAbstractionExtensions.cs] [C:\git\coverletissue\mytestlib\mytestlib.csproj]
C:\Users\Marco\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(24,5): warning : [coverlet] Unable to instrument module: C:\git\coverletissue\mytestlib\bin\Debug\net472\xunit.runner.reporters.net452.dll, embedded pdb without local source files, [C:\Dev\xunit\xunit\src\xunit.runner.reporters\Utility\EnvironmentHelper.cs] [C:\git\coverletissue\mytestlib\mytestlib.csproj]
C:\Users\Marco\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(24,5): warning : [coverlet] Unable to instrument module: C:\git\coverletissue\mytestlib\bin\Debug\net472\xunit.runner.utility.net452.dll, embedded pdb without local source files, [C:\Dev\xunit\xunit\src\xunit.runner.utility\Extensions\DisposableExtensions.cs] [C:\git\coverletissue\mytestlib\mytestlib.csproj]
C:\Users\Marco\.nuget\packages\coverlet.msbuild\2.6.3\build\coverlet.msbuild.targets(24,5): warning : [coverlet] Unable to instrument module: C:\git\coverletissue\mytestlib\bin\Debug\net472\xunit.runner.visualstudio.testadapter.dll, embedded pdb without local source files, [C:\Dev\xunit\xunit\src\xunit.runner.visualstudio\Utility\AssemblyExtensions.cs] [C:\git\coverletissue\mytestlib\mytestlib.csproj]
Test run for C:\git\coverletissue\mytestlib\bin\Debug\net472\mytestlib.dll(.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 16.2.0-preview-20190606-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 1,9011 Seconds

Calculating coverage result...
  Generating report 'C:\git\coverletissue\mytestlib\coverage.json'

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+
...

To hide the warning we expect that a user will add excludes, or we could move from warning to verbose. Feel free to close if solved!

[MarcoRossignoli]

I moved from warning to verbose because xunit will warning every time. https://github.com/tonerdo/coverlet/pull/548

[AArnott]

Thanks! The exclude workaround worked for me and I'll go with that for now.

[sharwell]

@MarcoRossignoli Why is it saying there is an embedded PDB without source files? The source files are part of the embedded PDB.

[MarcoRossignoli]

Maybe the message is not so clear...I mean "embedded pdb without local source files," local is about local hard drive...if sources are not on pc running test we skip.