search

stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.78k stars 185 forks source link

Unexpected coverage drop with 3.3.0 (and using the sln option) #2362

Closed riezebosch closed 1 year ago

riezebosch commented 1 year ago

Describe the bug From the normal coverage report is shows clearly that my SUT is covered, but still stryker thinks otherwise and refuses to kill some mutants.

Logs 2022-12-23.11-16-54.zip

Expected behavior A clear and concise description of what you expected to happen.

Desktop (please complete the following information):

Additional context Source project: https://github.com/riezebosch/CloudEventify

Liam-Rougoor commented 1 year ago

It looks like if I run Stryker on just the MassTransit project, it does mark the mutants as killed correctly. I'll investigate the solution run. image

Liam-Rougoor commented 1 year ago

@riezebosch Could you provide the logs for the Stryker run that does succeed? We noticed a warning where some projects 'extensions' could not be found: 2022-12-23T11:17:06.6775970+01:00 [VRB] Runner 2: [Warning] Could not find extensions: /Users/mriezebosch/git/CloudEventify/Rebus/CloudEventify.Rebus.IntegrationTests/bin/Debug/net6.0,/Users/mriezebosch/git/CloudEventify/MassTransit/CloudEventify.MassTransit.IntegrationTests/bin/Debug/net6.0,/Users/mriezebosch/git/CloudEventify/DaprApp.IntegrationTests/bin/Debug/net6.0,/Users/mriezebosch/git/CloudEventify/Dapr/CloudEventify.Dapr.IntegrationTests/bin/Debug/net6.0 (8f742cab) We would like to compare logs to see if we can find the issue there.

rouke-broersma commented 1 year ago

@Liam-Rougoor that warning comes from vstest and is expected.

yaelkeemink commented 1 year ago

We are having the same issue, In some cases (not all) Stryker does not seem to interpret the code coverage correctly. logfile.txt

Are there any updates on this issue?

causa-prima commented 1 year ago

I think I might encounter the same behavior:

Run against the complete solution, details for a specific project in it (dotnet stryker --open-report): Stryker_full_solution_run

Run for only the project from the screenshot above (cd to folder containing the test project, dotnet stryker --project Adapter --solution ..\<solutionfilename>.sln): Stryker_single_project_run

Because of confidentiality, I can't provide detailed logs. How else could I help? Or can you already reproduce the issue?

dupdob commented 1 year ago

there are several limitations in the current implementation for solution support. I have finalized a PR that should ensure solution runs and project runs get the same result. if you are willing to build it on your machine, you can check it, see if it solves your issue and report. #2400

riezebosch commented 1 year ago

stryker@3.7.1

[10:44:25 INF] 99 mutants created
[10:44:25 INF] Capture mutant coverage using 'CoverageBasedTest' mode.
[11:00:55 INF] 48    mutants got status NoCoverage.   Reason: Not covered by any test.
[11:00:55 INF] 26    mutants got status Ignored.      Reason: Removed by block already covered filter
[11:00:55 INF] 74    total mutants are skipped for the above mentioned reasons
[11:00:55 INF] 25    total mutants will be tested

stryker@improve_solution_handling

[11:18:16 INF] 99 mutants created
[11:18:16 INF] Capture mutant coverage using 'CoverageBasedTest' mode.
[12:28:02 INF] 18    mutants got status NoCoverage.   Reason: Not covered by any test.
[12:28:02 INF] 26    mutants got status Ignored.      Reason: Removed by block already covered filter
[12:28:02 INF] 44    total mutants are skipped for the above mentioned reasons
[12:28:02 INF] 55    total mutants will be tested

Looking good! No coverage on expected cases, like forcefully implemented overloaded interfaces and startup/program classes.

2023-04-04.11-03-29.zip

dupdob commented 1 year ago

thanks for the feedback

richardwerkman commented 1 year ago

This should have been fixed by @3.9.0