search

stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.76k stars 175 forks source link

False "NoCoverage" mark in the report #2251

Closed kant2002 closed 1 year ago

kant2002 commented 1 year ago

Describe the bug When run stryker over Refit I receive report where https://github.com/reactiveui/refit/blob/569108a65079d364647f06437f977df8145fc578/Refit/RestMethodInfo.cs#L68 is marked as NoCoverage. But if I remove this line, tests are fails.

That's puzzling.

Steps

  1. git clone https://github.com/reactiveui/refit
  2. cd refit/Refit.Tests
  3. dotnet stryker -p Refit.csproj

Expected behavior I would like that either mutation of this line was enabled, or something else what is meaningful from your perspective.

dupdob commented 1 year ago

It seems to be related to issue #2252. So I close this for now. Please reopen if this is unrelated.

kant2002 commented 1 year ago

No, if you notice that's different project for mutation. I assume that you mutate only project which I specify in command line. Please re-open

kant2002 commented 1 year ago

For the record, cannot reopen. It can be done only by you or another maintainer.

dupdob commented 1 year ago

Sorry for the mix up, I am looking into this

dupdob commented 1 year ago

Here are my findings:

  1. There seems to be a bug in the xUnit version used by the project (either xUnit.Core or xUnit.VisualStudio.Runner): it discards all test when provided with an empty filter string. I have implemented a fix that ensures Stryker provides null instead of an empty string.
  2. As this is a multi-targeting project, you should specify the target framework (net6.0 seems to be a good choice). As of today, Stryker selects the first properly defined framework of each project. This may lead to unexpected behaviour

The xUnit bugs demonstrate that Stryker does not handle this situation well and silently fails when tests are no longer detected after the initial testrun (dedicated to list available tests). What I find surprising is that no one discovered this problem before.

kant2002 commented 1 year ago

My first idea was to run dotnet stryker -p Refit.csproj -f net6.0 only to findout that -f is for configuration file. But at least mayby --framework can be passed from command line?

Time estimation is bad, it starts with 50 minutes, then drop to ~15 minutes and after that finished after 25 minutes.

Other than that, Refit provide approximately good results, some parts are not covered "by design", some things may be improved. I would say results are positive.

dupdob commented 1 year ago

Current time estimation has several flaws; the biggest one is that there is a significant initialization time that leads to overestimated run time when starting. There is also a bias because mutations with high coverage are run at the end, which may give an impression that things are slowing down. But, mutant coverage is also important: test runs are slower when the mutants survive. The purpose of this estimate is to give a rough idea of how long it will take