Multiple .NET projects and frameworks: incorrect file structure and coverage

[marthijn]

I am experiencing a problem with multiple projects and multiple build frameworks (C# .NET code). My project structure is as follows (link):

• /src/a/
• /src/b/
• /src/a.tests/
• /src/b.tests/

"a" is build against two target (.net standard and .net 8) "a.test" is build against two target (.net framework 4.8 and .net 8) "b" has a reference to "a" "b" and "b.test" are .net 8 only

The coverage output are 3 coverage files; a.net8.xml, a.net481.xml and b.xml. I sent these files using the files parameter. Then something strange happens. I see the files of "a" 3 times in the logs (where I expected 2, namely .net8 and .net standard). For example:

• "name": "home/runner/work/Sidio.OpenGraph/Sidio.OpenGraph/OpenGraph.cs",
• "name": "home/runner/work/Sidio.OpenGraph/Sidio.OpenGraph/OpenGraph.cs",
• "name": "home/runner/work/Sidio.OpenGraph/Sidio.OpenGraph/Sidio.OpenGraph/OpenGraph.cs", (this one is strange)

The file tree looks strange as well (duplicate files), and therefore the coverage calculation is wrong:

What am I doing wrong?

GIthub actions:

- name: Test with dotnet
  run: dotnet test --configuration Release
    /p:CollectCoverage=true
    /p:CoverletOutputFormat="\"cobertura,json\""
    /p:CoverletOutput=coverage/
    /p:IncludeTestAssembly=false
- name: Upload dotnet test results
  uses: coverallsapp/github-action@v2.3.0
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    files: ${{ github.workspace }}/src/Sidio.OpenGraph.Tests/coverage/coverage.net8.0.cobertura.xml ${{ github.workspace }}/src/Sidio.OpenGraph.Tests/coverage/coverage.net481.cobertura.xml ${{ github.workspace }}/src/Sidio.OpenGraph.AspNetCore.Tests/coverage/coverage.cobertura.xml
    format: cobertura
    base-path: ${{ github.workspace }}
    debug: true
  if: ${{ always() }}

[afinetooth]

Hi @marthijn.

What's happening is that you are uploading all three files at once. In that scenario, Coveralls will merge all the coverage reports in your upload, which in some circumstances could be fine, but it sounds like you would benefit more from sending one coverage report for each of your test runs (for each context).

And that is the typical use case for a project where you're testing your code in different contexts, such as across a matrix of different OS or framework versions.

One benefit of doing this is that Coveralls will separate out each report as a Subproject (aka. Job) in your Coveralls builds, letting you see their results and examine them separately.

As for the issue of duplicate files in your SOURCE FILES table: I think that is probably due to your setting of the base-path input option. That input option is canonically used on a once-per-upload basis because it's meant to correct for path discrepancies between the files listed in one coverage report and the same files' locations in your git repo (relative to your git root directory).

So, if you are using a single base-path value for three (3) different coverage reports generated for different subprojects in your repo, then at least two (2) of those adjustments will probably misalign and lead to the same files being treated as different files. because their paths are different.

Summary & next steps:

I think you'd benefit from sending separate coverage reports to Coveralls in a matrix configuration---which we refer to as a parallel build.

Steps:

1. Upload separate coverage reports - Rearrange your CI workflow to upload several coverage reports in several steps, in a matrix.
2. Follow this Parallel Build Usage Example - Since using a matrix build results in uploading several, sparate coverage reports, you'll then be sending what we call a parallel build. This usage example from the Coveralls GitHub Action README should be a perfect example because it is expressly for a parallel build configured as a matrix.
   • Make sure you use the input options shown in the example (although the carryforward option is less of a requirement than the others---more nice-to-have).
   • In addition to those input options, continue using the format option (format: cobertura). But since there should only be one coverage report per upload, the file, or files option is less important, so you might try leaving it out, at least at first.
   • And as for the base-path option, try leaving that out at first too, until it's proven that there is a discrepancy between your coverage report(s) and your git repo that the integration can't correct for. Then we can try to find the correct value for base-path for each upload step, which should be different for each case.

[marthijn]

Hi @afinetooth ,

Thanks for your detailed answer! Because a dotnet build or dotnet test command will be executed on any target framework found in the solution (my strategy only includes .net 8.0.x), I would expect a merge of the coverage reports to go fine. Just like it does in an IDE like Rider. Coveralls works differently here?

I will of course try your suggestion to rearrange the CI, and let you know the outcome. Thanks again!

[afinetooth]

Hi, @marthijn.

You're welcome!

Because a dotnet build or dotnet test command will be executed on any target framework found in the solution (my strategy only includes .net 8.0.x), I would expect a merge of the coverage reports to go fine. Just like it does in an IDE like Rider. Coveralls works differently here?

No, it shouldn't. Coveralls will simply take any coverage reports it finds in your CI environment and merge them together into a single report. As long as the filenames of individual files are identical across those reports, Coveralls will merge those into the coverage stats for a single file.

So the coverage stats for src/init.js could some from three reports present in a single upload, and will all get merged into a single "source file" coverage report.

However, if any/each of those coverage reports have a different path to the same file, because where the tests were run relative to the source file's location was different, for instance:

• Report 1: src/init.js
• Report 2: subproject_1/src/init.js
• Report 3: /init.js

Then those will end up as three (3) separate source file coverage reports, because the files will be considered different files.

An upload step can only take one corrective base-path: input, which will be applied to all reports in the upload. So if you have different paths to the same file across different reports, and you want to correct those, you will have to upload the reports individually.

One approach is to, first, upload all reports generated from a test run, as artifacts, then iterate through each with a different upload step in your CI config.