Selecting cobertura output causes coverlet to think there are no modules

[idg10]

Obviously, this issue doesn't affect all projects, as otherwise you'd have noticed before...

We have exactly one project that exhibits this problem. It's open source, so you can try it yourself. Here are the steps:

1. Clone https://github.com/corvus-dotnet/Corvus.Extensions.Newtonsoft.Json
2. cd into the Solutions folder
3. If you have .NET Core SDK 3.1.200 installed, you'll need to set an environment variable MSBUILDSINGLELOADCONTEXT=1 to work around a bug in the version of SpecFlow we're using (older 3.1.x SDKs don't need this; we see the coverlet problems on both 3.1.102 and 3.1.200 btw)
4. Run tests with default output dotnet test Corvus.Extensions.Newtonsoft.Json.sln -c Release /p:CollectCoverage=true
5. Run tests with coberatura output: dotnet test Corvus.Extensions.Newtonsoft.Json.sln -c Release /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura

The first test run works fine. We see this output:

+-----------------------------------+--------+--------+--------+
| Module                            | Line   | Branch | Method |
+-----------------------------------+--------+--------+--------+
| Corvus.Extensions.Newtonsoft.Json | 81.17% | 71.11% | 93.54% |
+-----------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 81.17% | 71.11% | 93.54% |
+---------+--------+--------+--------+
| Average | 81.17% | 71.11% | 93.54% |
+---------+--------+--------+--------+

The Corvus.Extensions.Newtonsoft.Json.Specs\coverage.json file is generated and looks to be fine.

But when running with the /p:CoverletOutputFormat=cobertura we get this:

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 100% | 100%   | 100%   |
+---------+------+--------+--------+
| Average | NaN% | NaN%   | NaN%   |
+---------+------+--------+--------+

and Corvus.Extensions.Newtonsoft.Json.Specs\coverage.cobertura.xml contains this:

<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="1" branch-rate="1" version="1.9" timestamp="1584600472" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0">
  <sources />
  <packages />
</coverage>

Apparently something about /p:CoverletOutputFormat=cobertura causes it to think that there are no modules being tested.

This happens building locally on Windows. You can see the exact same problem also happens on our CI build at https://dev.azure.com/endjin-labs/Corvus.Extensions.Newtonsoft.Json/_build/results?buildId=3522&view=logs&j=ca395085-040a-526b-2ce8-bdc85f692774&t=c58e8551-6b20-5658-7a75-0283b3f97acb which uses a Linux hosted build agent. Our CI build is currently set up to use .NET Core SDK 3.1.102 by the way, in order to work around the aforementioned problem with SpecFlow.

If we pass -c Debug we don't see the problem, but while I know you recommend running coverage against debug builds, we want our CI builds to verify that our code works in release builds, so we don't really want to switch over to running test against the debug config.

[idg10]

I've now run both these builds with verbose logging enabled, and when the /p:CoverletOutputFormat=cobertura option is present I see an extra line of output from the Coverlet.MSbuild.Tasks.InstrumentationTask. Here's the output when using the default format:

[coverlet] Included module filter 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.25.28610\ATLMFC\include;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.25.28610\include;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt'
[coverlet] Instrumented module: 'C:\dev\endjinoss\corvus-dotnet\Corvus.Extensions.Newtonsoft.Json\Solutions\Corvus.Extensions.Newtonsoft.Json.Specs\bin\Release\netcoreapp3.1\Corvus.Extensions.Newtonsoft.Json.dll'

And here's what I see when

[coverlet] Included module filter 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.25.28610\ATLMFC\include;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.25.28610\include;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt'
[coverlet] Unable to instrument module: C:\dev\endjinoss\corvus-dotnet\Corvus.Extensions.Newtonsoft.Json\Solutions\Corvus.Extensions.Newtonsoft.Json.Specs\bin\Release\netcoreapp3.1\Corvus.Extensions.Newtonsoft.Json.dll, pdb without local source files, [C:\dev\endjinoss\corvus-dotnet\Corvus.Extensions.Newtonsoft.Json\Solutions\Corvus.Extensions.Newtonsoft.Json\obj\Release\netstandard2.0\NuGet\0B88D08B5185509E90E3612AD5A962B592832014\Nullable\1.2.1\Nullable\NullableAttributes.cs]

The Nullable NuGet package enables the use of the attributes associated with C# 8.0's Nullable References feature in projects that target older frameworks such as netstandard2.0. It works by adding a source file into projects targetting such frameworks that defines local versions of the attributes. (The C# compiler doesn't care where the definitions of these attributes are, it just matches by fully-qualified name, so this works.)

So apparently when passing /p:CoverletOutputFormat=cobertura when the InstrumentModules target runs the Coverlet.MSbuild.Tasks.InstrumentationTask, that is attempting to instrument the Nullable.cs file that came from that Nullable NuGet package, and it's failing because it's not there.

As far as I can tell, the build process creates that file during the build process and deletes it again afterwards. The mystery, though, is why this works fine when you don't pass /p:CoverletOutputFormat=cobertura, and it also works fine if you do pass that and you also specifiy -c Debug.

[idg10]

So it looks like this is something you've seen before: https://github.com/tonerdo/coverlet/issues/714#issuecomment-579704498

But that has been marked as as-designed. Not sure that can be right. Source-only NuGet packages are relatively common. (And specifically, that Nullable NuGet package is widely used because it's the standard way to add support for C# 8.0 nullable references to libraries today without having to drop support for pre-.NET Core 3.1 frameworks.) Having to add a property to exclude content of this kind from coverage is not ideal.

So unless there's something wrong with the way the Nullable package in particular includes this content (in which case they need to fix that) this seems like something that just needs to work

[idg10]

A weird new discovery: it turns out that if I run this command repeatedly:

dotnet test Corvus.Extensions.Newtonsoft.Json.sln -c Release /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura

then it alternates between the right and wrong behaviour!

This appears to be an artifact of the way the IncrementalClean target in the SDK's Microsoft.Common.CurrentVersion.targets file works. It looks for an obj\<config>\<framework>\<ProjectName>.csproj.FileListAbsolute.txt file. Apparently this is a list of all the files produced by the preceding build. If the last build says that it produced a content asset file (e.g., Nullable.cs in this case), then the next build that runs will decide to delete that.

This problem occurs in the builds where the required file had been there from an earlier build and now gets removed.

[idg10]

Created https://github.com/dotnet/sdk/issues/10914 to report the alternating behaviour described in the preceding comment.

[MarcoRossignoli]

For now tracked as bug due to external issue, I mean It's unexpected that someone remove build source, it's expected that someone inject new sources that we need to skip. This is a new scenario for coverlet. As I've said we need to understand if we can be more relaxed on auto-exclusion.

Issue apart, if possible you should avoid msbuild integration and use collectors one https://github.com/tonerdo/coverlet#vstest-integration-preferred-due-to-known-issue-supports-only-net-core-application

[Bertk]

The issue might be out-dated because new releases are available.

Please reopen the issue if the bug still exists and can be reproduced with latest preview from coverlet nightly build.

see Consume nightly build