Would like to measure unit test quality by ignoring uncovered mutations

[mpownby]

We'd like to leverage the thresholds and mutation score feature that already exists in stryker, but we already have a fairly robust system in place for enforcing code coverage, so having stryker mix surviving mutants and "NoCover" mutants together when calculating the mutation score is problematic for us. We would prefer to use stryker to only measure the quality of our existing unit tests.

(pull request will be submitted shortly)

[rouke-broersma]

Hi @mpownby

This would first need to be discussed over at https://github.com/stryker-mutator/mutation-testing-elements because that contains our shared understanding of mutation testing concepts.

See https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/ for our definitions.

The change you've made now would only change the score calculation from the perspective of the stryker.net cli output and not for example the html report as the calculations for the html report exist in mutation-testing-elements.

[mpownby]

Hi @mpownby

This would first need to be discussed over at https://github.com/stryker-mutator/mutation-testing-elements because that contains our shared understanding of mutation testing concepts.

Ok, so what do I need to do? Is this a discussion between domain experts?

See https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/ for our definitions.

The change you've made now would only change the score calculation from the perspective of the stryker.net cli output and not for example the html report as the calculations for the html report exist in mutation-testing-elements.

I just tested using "-o:html" from command line and I got the behavior I want (the 'No Cover' column is all 0).

[mpownby]

NOTE I am not proposing to change any definitions. My proposal is more along the lines of a special ignore rule that gets applied as a result of a user action.

[rouke-broersma]

Sorry didn't read the change correctly, this shouldn't need a change in definitions. We'll discuss this internally.

[richardwerkman]

So, this would basically treat NoCoverage status as Ignored? Reducing the number of total mutants.

[mpownby]

So, this would basically treat NoCoverage status as Ignored? Reducing the number of total mutants.

Yes.

[rouke-broersma]

@mpownby We've discussed this and we have decided that this feature would not have sufficient common use and that it can easily be solved in some other way. Either by using exclusions or by for example by outputting the json report and modifying it with jq or something to change the statuses.

Sorry.