Different output format results in different coverage percentage

[cremor]

I would have assumed that the code coverage output format ist just that - a file format. But it seems like using a different output format also affects code coverage percentage. Is this by design?

Example: I use a .NET regular expression source generator in my code. When I use dotnet test with the argument --collect:"XPlat Code Coverage;Format=cobertura" then it does count the generated code in the namespace System.Text.RegularExpressions.Generated. When I use dotnet test with the argument --collect:"XPlat Code Coverage;Format=opencover" then it does not count the generated code.

Package: coverlet.collector 6.0.0 .NET 7

[daveMueller]

@cremor thanks a lot for reporting this 🙏 . I would also assume the same as you and thus it could really be a bug. Would it be possible for you to provide a simple sample that lets us reproduce this? No obligation at all, it could just help to analyse this faster.

[cremor]

I don't have access to an IDE for the next few days, but I'll try to drycode a repro 😅

// MyRegexUsage.cs
using System.Text.RegularExpressions;

namespace MyNamespace;

public partial class MyRegexUsage
{
    [GeneratedRegex("abc|def")]
    private static partial Regex AbcOrDefRegex();

    public static bool IsAbcOrDef(string text)
    {
        return AbcOrDefRegex().IsMatch(text);
    }
}

// MyRegexUsageTests.cs
using Xunit;

namespace MyNamespace.Tests;

public class MyRegexUsageTests
{
    [Fact]
    public void Test()
    {
        MyRegexUsage.IsAbcOrDef("def");
    }
}

Run:

dotnet test --collect:"XPlat Code Coverage;Format=cobertura"
dotnet test --collect:"XPlat Code Coverage;Format=opencover"

After that, you could run both code coverage reports through ReportGenerator to generate a HtmlInline_AzurePipelines report for each. But I assume (I hope 😅) this step is not relevant for the repro and you could just look in the generated code coverage report files. You'd have to look for code coverage for the System.Text.RegularExpressions.Generated namespace. The Cobertura report will have counted lines in it, the OpenCover report will not.

[daveMueller]

Sorry, I can't really reproduce it with your sample. The reports both have the generated functions. Also the coverage summary seems to be the same. Branch coverage 46.66 and line coverage 68.35.

<!-- opencover -->
<Summary numSequencePoints="79" 
visitedSequencePoints="54" 
numBranchPoints="30" 
visitedBranchPoints="14" 
sequenceCoverage="68.35" 
branchCoverage="46.66"
maxCyclomaticComplexity="35" 
minCyclomaticComplexity="35" 
visitedClasses="6" 
numClasses="6" 
visitedMethods="9" 
numMethods="9" />

<!-- cobertura -->
<coverage line-rate="0.6835" 
branch-rate="0.46659999999999996" 
version="1.9" 
timestamp="1691876749" 
lines-covered="54" 
lines-valid="79" 
branches-covered="14" 
branches-valid="30">

Can I ask which version of report generator you were using? Maybe it really is just a display problem.

coverage.cobertura.xml.txt coverage.opencover.xml.txt

[cremor]

Thanks for the analysis. I'll test this again in a few days.

[cremor]

You are right. It looks like the two xml files contain the same results. But after running them both through ReportGenerator version 5.1.24.0 I get different reports (different line numbers, branch numbers, percentages, ...).

What I noticed is that for the Cobertura report ReportGenerator generated two errors with the same message:

2023-08-16T09:00:29: File 'C:\...\src\WebApi\System.Text.RegularExpressions.Generator\System.Text.RegularExpressions.Generator.RegexGenerator\RegexGenerator.g.cs' does not exist (any more).
2023-08-16T09:00:29: File 'C:\...\src\WebApi\System.Text.RegularExpressions.Generator\System.Text.RegularExpressions.Generator.RegexGenerator\RegexGenerator.g.cs' does not exist (any more).

But for the OpenCover report it only generated one of those errors:

2023-08-16T09:00:42: File 'C:\...\src\WebApi\System.Text.RegularExpressions.Generator\System.Text.RegularExpressions.Generator.RegexGenerator\RegexGenerator.g.cs' does not exist (any more).

But even though it shows those two error messages, the converted Cobertura report then shows the System.Text.RegularExpressions.Generated namespace including details like line counts and method names, but doesn't show any code for it. In contrast, the converted OpenCover report doesn't show the namespace at all. And I assume it recalculates the line numbers, percentages and so on to exclude it.

This is the command I used for ReportGenerator:

tools\reportgenerator.exe -reports:path\to\coverage.*.xml -targetdir:path\to\output -reporttypes:HtmlInline_AzurePipelines

Do you think this is a problem with ReportGenerator and that I should report it there?

[daveMueller]

I tried this today with ReportGenerator version 4.8.7.0 and there both files have the same output. With the most current version, 5.1.24.0, the generated reports differ like you described.

@danielpalme Can you check if this is an error with our coverage reports or did ReportGenerator's parsers change so that the Cobertura parser now displays generated classes whereas the OpenCover still doesn't.

OpenCover:

Cobertura:

You can find the reports attached to this https://github.com/coverlet-coverage/coverlet/issues/1519#issuecomment-1676447115.

[danielpalme]

I will have a look in september. Can you please share your Cobertura file?

In general ReportGenerator does not care about whether classes are generated or not. It just lists the classes that are available in the OpenCover/Cobertura file. If you want to exclude certain classes you can use a filter: E.g. -classfilters:-*Generated

[cremor]

@danielpalme You can find the Cobertura (and OpenCover) file here: https://github.com/coverlet-coverage/coverlet/issues/1519#issuecomment-1676447115

[danielpalme]

I just released version 5.1.25 of ReportGenerator. This version should also include the class System.Text.RegularExpressions.Generator for the OpenCover file.