Mutation tests fails on Ubuntu

[RussKie]

Describe the bug

I'm bolting mutation tests on to a project. I expect that project contributors will generally work on Windows but the CI will run on Ubuntu. The MT run successfully on Windows (locally), however those fail on an Ubuntu (both locally under WSL2 and on CI).

There are a bunch of errors like this:

Safe Mode! Stryker will try to continue by rolling back all mutations in method. This should not happen, please report this as an issue on github with the previous error message. (2b631064)

Logs The logs can be downloaded here

Desktop (please complete the following information):

• OS: Ubuntu 20.04/22.04
• Type of project: .NET 8
• Stryker Version: 3.9.2

Additional context

To repro locally:

git clone https://github.com/Azure/dotnet-extensions-experimental.git mt
cd ./mt
git switch mutationtests
./restore.sh
./build.sh --mutationTest --configuration Release --bl

[rouke-broersma]

I can reproduce this locally, looks like most errors are caused by missing or inaccessible Types. Eg 'Throw' is inaccessible due to its protection level. Maybe a referenced assembly or an msbuild target is missing during stryker reference resolving. Maybe InternalsVisibleToTest isn't working correctly for some reason in stryker on linux.

[dupdob]

there is another issue: there is no explicit package dependency to a testrunner in some of the test projects. As such, initial tests fail on windows so Stryker stop there. They work on Linux, probably because the xUnit.runner is part of the default adapters in the distribution.

[RussKie]

I can reproduce this locally, looks like most errors are caused by missing or inaccessible Types. Eg 'Throw' is inaccessible due to its protection level. Maybe a referenced assembly or an msbuild target is missing during stryker reference resolving. Maybe InternalsVisibleToTest isn't working correctly for some reason in stryker on linux.

Throw.cs is conditionally referenced in a project if InjectSharedThrow = true: https://github.com/Azure/dotnet-extensions-experimental/blob/b28a1decf1b59b223ba4612e9ec557c8e98af371/eng/MSBuild/Shared.props#L2-L4 InjectSharedThrow is set to true unless already set: https://github.com/Azure/dotnet-extensions-experimental/blob/b28a1decf1b59b223ba4612e9ec557c8e98af371/src/Directory.Build.props#L14

All logs (e.g., Microsoft.Azure.Extensions.AmbientMetadata.ServiceFabric.Tests/log-20230622.txt) contain the entry:

2023-06-22T13:12:04.8556489+00:00  [VRB] Property "InjectSharedThrow"="true" (b1bde488)

The IVT are set like this: https://github.com/Azure/dotnet-extensions-experimental/blob/b28a1decf1b59b223ba4612e9ec557c8e98af371/src/Microsoft.Azure.Extensions.AmbientMetadata.ServiceFabric/Microsoft.Azure.Extensions.AmbientMetadata.ServiceFabric.csproj#L27-L29

Is stryker changing the name of the test assembly?

[RussKie]

there is another issue: there is no explicit package dependency to a testrunner in some of the test projects. As such, initial tests fail on windows so Stryker stop there. They work on Linux, probably because the xUnit.runner is part of the default adapters in the distribution.

I've observed this too. And locally on a fresh clone it fails with the exact same error. However, on a re-run - it's working successfully. I'm still scratching my head...

[EDIT] I fixed that by settings DOTNET_ROOT and PATH env vars. https://dev.azure.com/dnceng-public/public/_build/results?buildId=317919&view=logs&j=1b63fd69-ee25-528d-27b4-7b1fc37fdf86&t=b889c08e-7d1e-5552-7255-3f78a875d921

[dupdob]

well, it looks like this projects depends to much on internal preview versions/component for me to be able to build it locally. I am not able to build it locally via VStudio, nor via Stryler internal MsBuild build. So I can't do much. I may try again in the future, but I am afraid I other stuff to attend to.

[RussKie]

this projects depends to much on internal preview versions/component for me to be able to build it locally. I am not able to build it locally via VsTest, nor via Stryler internal MsBuild build.

We're on the bleeding edge, but everything is (or should be) buildable locally. Can you please elaborate how you tried building it and what wasn't working?

[dupdob]

hi, thanks for the help. For information, I tried VS V17.2 (Preview 2) as well as the latest public Net 8 SDK. Had to change the hard coded framework version in the prop files, but still got a bunch of weird error related to Nuget resolutions.

I am not saying the problem is related to using pre release version, but this project has strong requirements in order to be buildable on my machine and I do not have enough time to fulfill them for the time being. I am busy with a VsTest related issue (hence me typing VSTest instead of Visual Studio in my previous response)

I may get back to this later, or maybe someone else on the team will.

[dupdob]

that being said, this issue stresses that Stryker should provide better reporting regarding tests failing initially.

[RussKie]

hi, thanks for the help. For information, I tried VS V17.2 (Preview 2) as well as the latest public Net 8 SDK. Had to change the hard coded framework version in the prop files, but still got a bunch of weird error related to Nuget resolutions.

Unless you have the required .NET 8 (that's declared in global.json) installed globally (i.e., in C:\Program Files\dotnet) opening the solution by double clicking on the sln file won't work. The solution can only be opened with start-vs.cmd which pulls down the required toolset and sets the necessary env vars.

That said, the issue is only observable on Linux (under stryker; without it the solution builds and tests run without any issues), on Windows everything is working.

[RussKie]

Can you help me understand how stryker builds the solution? Maybe I could help figuring out what's missing.

[rouke-broersma]

Can you help me understand how stryker builds the solution? Maybe I could help figuring out what's missing.

Stryker uses the solution and/or project files to resolve dependencies and the like. This is done using a tool called Buildalyzer. After that stryker places mutations using Roslyn. Then Stryker compiles the mutated sources in-memory using Roslyn here: https://github.com/stryker-mutator/stryker-net/blob/master/src/Stryker.Core/Stryker.Core/Compiling/CsharpCompilingProcess.cs#L60

The part where something is most likely going wrong is in Buildalyzer, we are most likely missing some msbuild targets or something during analysis which causes a loss of information during compile. The compile information is mostly gathered using Buildalyzer here: https://github.com/stryker-mutator/stryker-net/blob/master/src/Stryker.Core/Stryker.Core/Initialisation/Buildalyzer/IAnalyzerResultExtensions.cs

Thing such as:

• Compiler options (signing, Unsafe, Nullable etc)
• Parse options (Language version, preprocessor symbols etc)
• Source Generators

My best guess is that due to the way Buildalyzer analyses msbuild projects either a target or an msbuild variable is missing that's causing the errors.

[dupdob]

hi @RussKie, I just read your findings. As said earlier, I do not plan to try and reproduce for now. But I may have a couple of remarks that could help narrow the problem down.

1. Stryker does not mutate a project if the initial test(s) report 0 test case ==> it is likely that if the test problem is fixed, you will get the same error on windows. I think this is important, because there is no plausible reason for a difference depending on the OS
2. Stryker does not alter test assemblies in any way, except by building them, either via solution build (for solution mode) and dotnet build otherwise ==> this may be the cause of your problem

if this is 2, I think Stryker could offer an option not to build test assemblies. This is just a precaution to ensure those are up to date.

[RussKie]

2. Stryker does not alter test assemblies in any way, except by building them, either via solution build (for solution mode) and dotnet build otherwise ==> this may be the cause of your problem

if this is 2, I think Stryker could offer an option not to build test assemblies. This is just a precaution to ensure those are up to date.

If we could avoid rebuilding, this would be great. I take it this option isn't available today, it's a FR.

[rouke-broersma]

Many people have tried, but so far have been unsuccessful in making that happen due to the way buildalyzer works. We would have to replace Buildalyzer for test projects at the very least.