search

stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.75k stars 176 forks source link

Show code coverage in report #2662

Open ISErikMeijer opened 10 months ago

ISErikMeijer commented 10 months ago

If possible I would like if it showed the normal overall code coverage in the final stryker report, so I don't have to separately run a code coverage test.

dupdob commented 10 months ago

That would be indeed beneficial. Sadly, mutation coverage is very poor proxy for code coverage as it focuses on where Stryker injected mutations with no concern about non mutated code. Furthermore, we capture coverage for a mutated version of the project, not the original one. As such: 1) metrics would be wrong as Stryker adds to the code base 2) coverage could be wrong (in subtle way) for some test/mutated code interferences.

In short, this is an interesting objective, but I do not see this happening in a near or medium future

dupdob commented 10 months ago

that being said, the mutation coverage data can be published. But, again, this is a different report.

richardwerkman commented 10 months ago

We might be able to reuse the coverlet collector https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.collector/DataCollection/CoverletCoverageCollector.cs

dupdob commented 10 months ago

We could capture code coverage as part of the initial test run (that could be used as a normal test run BTW), but no use benefit beyond that. I am probably missing something important, but:

  1. As pointed above, code coverage results for mutated code may not reflect coverage for normal code
  2. Coverlet does not capture coverage per test, only per session. This would significantly impair coverage based optimization, as we could only determine which mutants are tested, without knowing which test(s) cover(s) them

There are also significant hurdle to address

  1. Ensure assembly PDBs contain location information for mutation (as of today, mutated code bears no location information)
  2. Parse coverage result to determine mutant coverage
richardwerkman commented 10 months ago

@dupdob I don't think we need to use the coverage information to determine mutant coverage. We can keep that like it is. We just need to add coverage analysis to the initial testrun and get the percentage from it. The initial testrun doesn't contain mutations so the coverage should be correct.

dupdob commented 10 months ago

oh, ok. I feel better then :-)