Improve handling of xUnit theories in coverage analysis

[dupdob]

Is your feature request related to a problem? Please describe. Stryker run mutations with dubious coverage info against all tests. This is a logical defensive measure, but it can be a problem when there are many of those mutants. The main cause of 'dudbious' coverage comes from some VsTest/xUnit shortcomings: some test Theories are reported as executed between a testcase end and testcase start events.

Describe the solution you'd like Stryker could find a way to properly associate those results with the proper test and thus, limits the number of 'dubious coverage' mutants.

Alternative solution An alternative solution could be to perform isolated coverage analysis for the problematic tests. This would provide more accurate results but would increase run time

Additional context It is likely this feature will require some heuristically based 'guessing' so it should be configurable in case it provides the wrong guess, it should be possible to turn it off. May via a new mode for coverage analysis. In practive, with xUnit, 'dubious' mutants appear either before the test (when using a runtime datasource) or after the test (when using InlineDataSet); when mutants appear between tests, Stryker could associate those with the test that has several results (either the one before or the one after). When both tests are likely candidates, revert to legacy mode.

[dupdob]

As a reminder, here is a list of known cause of problems for coverage analysis (coverage discovery actually):

1. TestTheories :
   1. if the testrunner cannot distinguish them, they are linked to the same test case. This means the coverage for the result after the first one will leak to the next test. As of V1.51 we flag the mutants covered by those tests as to be tested against every test as a precaution.
   2. If the theory is the last or sole test, the coverage info is simply lost for now. The Stryker data collector identifies the situation, but we have yet to find a way to report the data(*).
   3. the testrunner must wait for all results for a testcase before deciding if it passes (but it can mark is as failed as soon as it gets a failed result)
2. MemberData or ClassData:
   1. the test parameters are enumerated before the testcase starts, so any mutants covered during this phase will appear as leaking from the previous test. At least, we always report the situation, even if it happens before the first test (the Datacollector report on TestCaseEnd events)
   2. mutation(s) can change how many data sets are generated, which makes tracking the result even more difficult
   3. if such a test results in a single testcase, it adds all problem listed above
   4. if such a test results in multiple testcases, all mutant hits related to data discovery will be reported before the first test case (*). The proper approach there is probably to add all the testcases as covering those mutants, but it appears to be hard to achieve.

After a bit of thinking, I am going to pursue the alternative approach, by capturing separately the coverage for test with multiple results. This should tackle most issues listed above, excluding the one marked with(*). Based on reported issues, Theories are present in most test bases. Working on this should improve performance for most (xUnit) projects. I hope to increase reliability along the way, but not sure how at this stage

[dupdob]

Trying this, I realized I made a mistake: I obviously slam myself against not being able to get data from the extra results when those tested are run isolated (item 1.ii in my previous comment).

Based on logs, it seems that the this problem between test cases and test results is due to some VsTest or VsTest wrapper shortcomings, as the log report provides different identification for each test result!

[dupdob]

I put this issue on hold until the potential new testrunner is integrated. (issues listed here are linked to current VsTest architecture and could be solved by MsTest)