stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.76k stars 177 forks source link

Incorrect Compilation Failure causes fallback to Safe Mode #2666

Closed JackSteel97 closed 1 year ago

JackSteel97 commented 1 year ago

Describe the bug Stryker throws the below warning logs in a few places involving nullable types that have been defaulted with the ?? operator. The line mentioned in the logs below looks like this:

 builder.WithStartTime((options.StartTime ?? DateTime.MinValue).ToUniversalTime());

Logs

[14:02:07 WRN] Stryker.NET encountered a compile error in C:\...\ClaimsDataCollector.cs (at 53:214) with message: 'DateTime?' does not contain a definition for 'ToUniversalTime' and no accessible extension method 'ToUniversalTime' accepting a first argument of type 'DateTime?' could be found (are you missing a using directive or an assembly reference?) (Source code: ToUniversalTime)
[14:02:07 WRN] Safe Mode! Stryker will try to continue by rolling back all mutations in method. This should not happen, please report this as an issue on github with the previous error message.

Expected behavior Compilation succeeds because the actual type being operated on by ToUniversalTime is a never null DateTime

Desktop (please complete the following information):

dupdob commented 1 year ago

thanks for reporting this; in short, this is not an issue. Let me explain:

Compilation succeeds because the actual type being operated on by ToUniversalTime is a never null DateTime

Compilation failed because Stryker injected one mutation that broke this guarantee (never being null); the fact that it results in a compilation error means the mutation was killed by the compilation phase (no test needed). So far so good.

The problem is that Stryker cannot identify which mutations triggered this compilation error, so it marks all mutations within the method as being killed by the compilation phase. That is why you have this warning.

JackSteel97 commented 1 year ago

If this is expected behaviour then maybe the This should not happen, please report this as an issue on github with the previous error message. part of the warning message should be removed?

rouke-broersma commented 1 year ago

If this is expected behaviour then maybe the This should not happen, please report this as an issue on github with the previous error message. part of the warning message should be removed?

It is expected but we also want to be notified that this happend in case we can change something to avoid the scenario. The explanation is mostly to let you know you don't need to worry about this.

dupdob commented 1 year ago

just to be very precise:It is expected that some mutations will result in compilation errors. Stryker is often able to link a compilation error to the culprit. Which is then removed and marked as compilation error, and compilation will be successful. This is not reported as a warning.

There is a warning if and only if Stryker is not able to identify which mutation triggers the compilation error, it has to remove every mutations (in that method) for the compilation to be successful. This is far from ideal, hence the warning.