How to merge reports?

[RussKie]

Is your feature request related to a problem? Please describe.

Stryker generated a report per project, e.g., in our case we get two files: mutation-report.html and mutation-report.json. We use Mutation Report Publisher to publish the HTML report. However, if a solution contains multiple mutated projects the report tab looks something like this:

Obviously for a solution that may contain dozens (or hundreds) of projects this won't scale and presents a usability issue.

I don't think I'm the only one in need of this functionality, e.g., others have to resort to some kind of manual merge too (see https://github.com/stryker-mutator/stryker-net/issues/1676).

Describe the solution you'd like

I'd like to know how the reports can be merged together. I can put together a script, but if this can be facilitated by stryker itself - it'd be grand. Something like dotnet stryker merge **/mutation-report.json --output ./merged-mutation-report.html (taking an inspiration from dotnet-coverage merge).

[rouke-broersma]

Hi @RussKie

Currently we support delivering a merged report in Solution Mode: https://stryker-mutator.io/docs/stryker-net/operating-modes/#solution-file-context

Or when using the stryker dashboard: https://stryker-mutator.io/docs/stryker-net/reporters/#dashboard-reporter when supplying the individual projects as modules: https://stryker-mutator.io/docs/stryker-net/configuration/#project-infomodule-string

[RussKie]

Unfortunately, we can't use any of these options. How can I merge the files manually?

[rouke-broersma]

Unfortunately, we can't use any of these options. How can I merge the files manually?

Should be as simple as merging the files and testFiles properties. The html report will do the rest. @psfinaki probably has something for this for you to use that's already in use within Microsoft.

[patwalb]

Unfortunately, we can't use any of these options. How can I merge the files manually?

It is actually quite simple. I tested it. Each JsonFile has following attributes:

• SchemaVersion
• ProjectRoot
• Thresholds
• Files
• TestFiles

Files and TestFiles have a lot of nested content, but it does not matter, if you merge correctly.

Given you have report1.json and report2.json-> You just merge it to

{
schemaVersion: <value_schemaVersion_report1> (if both schemaVersion are not equal -> Error),
thresholds: <value_thresholds_report1> (if both thresholds are not equal -> Error),
projectRoot:[<value_projectRoot_report1>, <value_projectRoot_report2>],
files: {<value_files_report1>, <value_files_report2},
testFiles: {<values_testFiles_report1>, <values_testFiles_report2>}
}

[psfinaki]

@patwalb yes, correct. You can take inspiration here, although it needs a tweak to merge test files as well.

[richardwerkman]

I suggest we implement this as follows:

dotnet stryker merge **/mutation-report.json another-report.json --output ./merged-mutation-report.html

For this we can use multi-value arguments.

This can be built completely in the CLI project, without having to modify the stryker.core project.

[rouke-broersma]

I'm not sure stryker is the correct place for this. Sounds more like a good suggestion for the widely used https://github.com/danielpalme/ReportGenerator to add support for the mutation testing report schema as input/output.

[RussKie]

I disagree. It's the stryker's format, and the stryker itself is in the best position to know how to merge its reports. Just like code-coverage tool: https://learn.microsoft.com/dotnet/core/additional-tools/dotnet-coverage#dotnet-coverage-merge.

[rouke-broersma]

I disagree. It's the stryker's format, and the stryker itself is in the best position to know how to merge its reports. Just like code-coverage tool: https://learn.microsoft.com/dotnet/core/additional-tools/dotnet-coverage#dotnet-coverage-merge.

You're linking a tool that is external from dotnet-cli or visual studio, so I don't see much difference. The output format used by stryker is an open format with the schema available and it's used by multiple mutation testing tools. We could provide a merge tool with the schema sure, but there already is a tool that is widely used for the purpose of merging reports and they have indicated to be interested in supporting the mutation testing schema.

If were were to create something like this from the stryker teams it would not be built into stryker.net since it would be useful for all mutation testing tools under the stryker name and all mutation testing tools that support the same open schema.