dotnet-coverage tool feature request: Arguments to support multiple output files when using multiple output formats.

[josundt]

The dotnet-coverage tool today supports multiple coverage output formats and output files when using a comma-separated list in the -f, --output-format argument . This works even if it is not a well-documented feature.

Supporting multiple output formats is useful since different coverage formats are typically required by various tools and services used post test execution. In our case we need different output formats for Azure DevOps coverage reports (Cobertura works best here); but we also need to upload coverage reports to SonarQube/SonarCloud (OpenCover works best).

Remark: Both Azure DevOps and SonarQube now support the Microsoft XML format that dotnet-coverage can output. I found two problems with this format, and we can not use it for these two reasons:

1. It looks like this XML format does not contains line coverage data - after importing this format across our different tools, the reported line coverage is totally different in each tool. A simple test showed that while 6 lines were reported as covered inside Visual Studio, Azure DevOps reported 4 lines, and SonarQube reported 2 lines. When using Cobertura/OpenCover, the line coverage metrics have been identical across Azure DevOps and SonarQube. SonarQube uses line coverage as them main coverage metric, so when switching from OpenCover to XML, line coverage percentage dropped to approximately half of what it had been.

2. Azure DevOps creates better and more detailed HTML reports from a Cobertura file than from the MS XML file).

The current limitation with the dotnet-coverage tool is that the -o, --output argument does not work in combination with multiple output formats (comma-separated list). You can't control output folder or output file names when multiple output formats. When omitting this argument, files in different formats are written to current working directory (with the file base name "output" followed by format specific suffix + extension).

The workaround I have used for now is to change current working directory before executing the tool. Still I don't find this to be intuitive. The tool needs a parameter that allows specifying path for output files or output folder when using multiple output formats.

Until this is in place, coverlet remains the preferred option for us.

[fhnaseer]

It looks like this XML format does not contains line coverage data - after importing this format across our different tools, the reported line coverage is totally different in each tool. A simple test showed that while 6 lines were reported as covered inside Visual Studio, Azure DevOps reported 4 lines, and SonarQube reported 2 lines.

Can you please share what are the difference from code perspective?

[josundt]

@fhnaseer I have not studied the MS Coverage XML syntax deeply, but as mentioned the LINE coverage statistics changed drastically after switching from OpenCover for SonarQube and Cobertura for Azure DevOps PCCR v2 (Publish code coverage results) to MS Coverage XML for both.

My hypothesis was that the MS Coverage XML format maybe lacks LINE coverage data; and as a result the LINE coverage computation (based on BRANCH coverage metrics and source files) is left with each tool that imports the coverage files (SonarQube/Azure DevOps).

And when LINE coverage computation is done differently in these different tools, the metrics do not align.

In a simple test I did using MS Coverage XML, Line coverage reported in Azure Devops PCCR v2 was 4 lines; in SonarQube it was 2 lines. Not sure if my assumptions may be correct?

For now I needed to switch back to OpenCover/Cobertura

[jakubch1]

@josundt could you please provide coverage reports (OpenCover/Cobertura and MS XML) with information for which exact line you see difference?

[josundt]

@jakubch1 I have not examined the XML output file; I don't the think your written XML file necessarily contains incorrect data. I'll give you some more details about my findings:

1. In the Azure DevOps pipeline after switching from Cobertura (using Publish Code Coverage Results v1 task) to MS Coverage XML (using Publish Code Coverage Results v2 task), the reported "lines covered" remained exactly the same 👍

2. With SonarCloud after switching from OpenCover to MS Coverage XML, the "lines covered" count dropped by 50%.

I looks like the main problem may be SonarCloud rather than the XML file written by your tool. Still the only currently acceptable solution for us is to revert back to OpenCover and coverlet.

If your tooling had supported the OpenCover output format (and ideally also an improved way specifying output file or folder paths when using multiple files in different formats), we would have started to use Microsoft.CodeCoverage instead.

[jakubch1]

@josundt could you please check if this is related to: #101. Basically please check if report generated by MS Coverage contains /_/ is source file paths.

[josundt]

I have checked and these issues are unrelated. In my scenario I am neither merging files nor using using cobertura format at all. I am simply generating XML output format using dotnet-coverage collect.

As mentioned earlier, I don't the think data in the output file (xml format) contains incorrect data; the problem is more likely that coverage parsers in SonarCloud are inconsistent, and line coverage is 50% less when parsing MS XML compared to when parsing OpenCover.

Regardless; -for this reason we needed to revert back to using coverlet that supports the OpenCover format. If you could also support the OpenCover format (which would get you closer to feature parity with coverlet), we could have switched to using Micrsoft.CodeCoverage.

[jakubch1]

OK, in such case I am closing this as we have another issue for OpenCover format.