[BUG] coverage.cobetura.file.xml filenames inconsistent between .NET 7 and .NET 8

[groogiam]

Describe the bug There appears to be an pretty severe inconsistency between the cobetura output in .NET 7 and .NET 8. Please see issue below for details.

To Reproduce See https://github.com/microsoft/azure-pipelines-tasks/issues/19901 for the difference in output between .NET 7 and .NET 8

Expected behavior Output should be consistent across runtimes. Or at a minimum there should be a flag to have .NET 8 output the full path to the file.

Actual behavior See linked issue.

Configuration (please complete the following information): Please provide more information on your .NET configuration:

• Which coverlet package and version was used? 6.0.0
• Which version of .NET is the code running on? . SDK 8.0.300 and 7.0.409
• What OS and version, and what distro if applicable? Windows
• What is the architecture (x64, x86, ARM, ARM64)? X64
• Do you know whether it is specific to that configuration?

Additional context Add any other context about the problem here.

:exclamation: Please also read Known Issues

[tungi52]

Hi! I have exactly the same issue with an extra twist. After switching .NET 8 the filenames changed from c:\users\me\...\ProjectA\ClassA.cs to https://github.com/.../ProjectA/ClassA.cs. The additional problem on my side is that these filenames are both present in the same coverage.cobertura.xml.Most of them is the new https version but some of them still refer to my filesystem. Unfortunately we use some futher tools to process these files (for example reportgenerator) and the calculated coverage is incorrect due to a inconsistent filenames.

I do some research and I found the following:

1. New filenames: it is the result of a breaking change in .NET 8 more information can be found here and here. So in this case I think when UseSourceLink option is turned on this is a normal behaviour. However in this method https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.core/Reporters/CoberturaReporter.cs#L216 when source links are disabled it returns only the filename (probably it is correct) but comparing to previous .NET versions I have the feeling that source links are mixed up with full physical paths. I don't know but maybe a clarification would be great.

2. About the mixed filenames: After a few test and detailed logging I think I found the problem: https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.core/Coverage.cs#L368 between line 368 and 387. If there is no hits for an assembly the sourcelink is not used but I think this is not a correct or expected behaviour. If we reference a not covered assembly but would like to use sourcelinks the filename attribute should refer to them as well in the output file. A naive solution could be to change the order of the two if condition but I'm not familier with this project so I don't know what could be the consequences.

[Bertk]

Please use latest coverage.collector V6.0.2 package and replicate the issue.

[tungi52]

@Bertk I used the latest version.

[tungi52]

@Bertk I set up a small project which demonstrates the issue: https://github.com/tungi52/CoverletBug

In the TestResults folder you can see the problem. You can also reproduce the issue just clone the repo then change the SDK version in the global.json file (not forget to install both .NET6 and .NET8 SDK on your system) and run either

dotnet test CoverletBug_net6.sln --collect:"XPlat Code Coverage" --settings TestProject/test.runsettings

or

dotnet test CoverletBug_net8.sln --collect:"XPlat Code Coverage" --settings TestProject/test.runsettings

Hope this helps.

[Bertk]

Thank you for the repo but I was not able to reproduce the issue on my system. I noticed some generated code in your TestResults/with_sdk6/coverage.cobertura.xml file which is not available on my system.

I used the following sequence to create the output in artifacts.zip.

dotnet restore C:\GitHub\coverlet-issue-1658\CoverletBug_net6.sln
dotnet build C:\GitHub\coverlet-issue-1658\CoverletBug_net6.sln
dotnet test TestProject\TestProject_net6.csproj -c Debug --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura
reportgenerator -reports:"TestProject/**/*.cobertura.xml" -targetdir:"artifacts\reports.cobertura.net6.0" -reporttypes:"HtmlInline_AzurePipelines_Dark;Cobertura"

dotnet clean
rd /S /Q TestProject\TestResults
=> "adapt global.json for net8.0"

dotnet restore C:\GitHub\coverlet-issue-1658\CoverletBug_net8.sln
dotnet build C:\GitHub\coverlet-issue-1658\CoverletBug_net8.sln
dotnet test TestProject\TestProject_net8.csproj -c Debug --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura
reportgenerator -reports:"TestProject/**/*.cobertura.xml" -targetdir:"artifacts\reports.cobertura.net8.0" -reporttypes:"HtmlInline_AzurePipelines_Dark;Cobertura"

artifacts.zip

[tungi52]

@Bertk did you clone my repo or just download it? Because the issue is reproducable only in the first case.

[Bertk]

I cloned the repository today and used the configuration combinations Format&SingleHit, Format&UseSourceLink, Format&IncludeTestAssembly, Format&SkipAutoProps.

I also used the provided test.runsettings file.

dotnet test TestProject\TestProject_net6.csproj -c Debug --collect:"XPlat Code Coverage" --settings TestProject\test.runsettings

I still cannot reproduce the issue on my system.

[tungi52]

the issue is only for .net 8. Did you try running this too on the cloned repo?

dotnet test TestProject\TestProject_net8.csproj -c Debug --collect:"XPlat Code Coverage" --settings TestProject\test.runsettings

[Bertk]

Finally, this issue is not related to cobertura coverage file format but the UseSourceLink option and coverlet.collector uses the local file name for net6.0 because UseSourceLink is not activated for net6.0 project files.

There is a .NET 8.0 breaking change Source Link included in the .NET SDK

The filename without activated UseSourceLink option.

    <package name="CoveredClassLibrary_net8" line-rate="1" branch-rate="1" complexity="1">
      <classes>
        <class name="CoveredClassLibrary.CoveredClass" filename="CoveredClassLibrary\CoveredClass.cs" line-rate="1" branch-rate="1" complexity="1">
          <methods>
            <method name="Sum" signature="(System.Int32,System.Int32)" line-rate="1" branch-rate="1" complexity="1">
              <lines>
                <line number="5" hits="1" branch="False" />
              </lines>
            </method>
          </methods>
          <lines>
            <line number="5" hits="1" branch="False" />
          </lines>
        </class>
      </classes>

The filename with activated UseSourceLink option.

    <package name="CoveredClassLibrary_net8" line-rate="1" branch-rate="1" complexity="1">
      <classes>
        <class name="CoveredClassLibrary.CoveredClass" filename="https://raw.githubusercontent.com/tungi52/CoverletBug/89244e69a571cda8ff30e53aa8aa4760e0985249/CoveredClassLibrary/CoveredClass.cs" line-rate="1" branch-rate="1" complexity="1">
          <methods>
            <method name="Sum" signature="(System.Int32,System.Int32)" line-rate="1" branch-rate="1" complexity="1">
              <lines>
                <line number="5" hits="1" branch="False" />
              </lines>
            </method>
          </methods>
          <lines>
            <line number="5" hits="1" branch="False" />
          </lines>
        </class>
      </classes>

Please

[tungi52]

Please aware that if a library is fully uncovered with UseSourceLink option the tool generates different filenames in the same report:

https://github.com/tungi52/CoverletBug/blob/main/TestResults/with_sdk8/coverage.cobertura.xml#L9

https://github.com/tungi52/CoverletBug/blob/main/TestResults/with_sdk8/coverage.cobertura.xml#L25

[Bertk]

@tungi52 Please create a separate issue for the inconsistent file name of uncovered library with enabled SourceLink configuration.

About the mixed filenames: After a few test and detailed logging I think I found the problem: https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.core/Coverage.cs#L368 between line 368 and 387. If there is no hits for an assembly the sourcelink is not used but I think this is not a correct or expected behaviour. If we reference a not covered assembly but would like to use sourcelinks the filename attribute should refer to them as well in the output file. A naive solution could be to change the order of the two if condition but I'm not familier with this project so I don't know what could be the consequences.

The proposed update could have an performance impact and need further investigations.

[Bertk]

Newer issue #1679 addresses inconsistent filenames in coverage files with activated SourceLink option.