Stryker failing with NullReferenceException

[richardwerkman]

Describe the bug When running stryker in solution mode, mutation fails with a null reference exception. I ran stryker on another clone of stryker (master branch).

[10:49:37 ERR] Some mutations failed to be inserted, they are dropped.
[10:49:37 ERR] Some mutations failed to be inserted, they are dropped.
[10:49:37 ERR] Some mutations failed to be inserted, they are dropped.
[10:49:40 INF] Time Elapsed 00:01:44.7349597
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.CodeAnalysis.CSharp.SyntaxFactory.FindConditionalAccessNodeForBinding(CSharpSyntaxNode node)
   at Microsoft.CodeAnalysis.CSharp.Binder.GetReceiverForConditionalBinding(ExpressionSyntax binding, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindMemberBindingExpression(MemberBindingExpressionSyntax node, Boolean invoked, Boolean indexed, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindValue(ExpressionSyntax node, BindingDiagnosticBag diagnostics, BindValueKind valueKind)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindValueConditionalOperator(ConditionalExpressionSyntax node, ExpressionSyntax whenTrue, ExpressionSyntax whenFalse, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindConditionalOperator(ConditionalExpressionSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindParenthesizedExpression(ExpressionSyntax innerExpression, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindValue(ExpressionSyntax node, BindingDiagnosticBag diagnostics, BindValueKind valueKind)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindConditionalAccessExpression(ConditionalAccessExpressionSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindSimpleBinaryOperator(BinaryExpressionSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindValue(ExpressionSyntax node, BindingDiagnosticBag diagnostics, BindValueKind valueKind)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindValueConditionalOperator(ConditionalExpressionSyntax node, ExpressionSyntax whenTrue, ExpressionSyntax whenFalse, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindConditionalOperator(ConditionalExpressionSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindParenthesizedExpression(ExpressionSyntax innerExpression, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, BindingDiagnosticBag diagnostics, Boolean invoked, Boolean indexed)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindValue(ExpressionSyntax node, BindingDiagnosticBag diagnostics, BindValueKind valueKind)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindBooleanExpression(ExpressionSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindIfStatement(IfStatementSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindStatement(StatementSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindBlockParts(BlockSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindBlock(BlockSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindStatement(StatementSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindPossibleEmbeddedStatement(StatementSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindIfStatement(IfStatementSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindStatement(StatementSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindBlockParts(BlockSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindBlock(BlockSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindStatement(StatementSyntax node, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindMethodBody(CSharpSyntaxNode declaration, BlockSyntax blockBody, ArrowExpressionClauseSyntax expressionBody, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindMethodBody(CSharpSyntaxNode syntax, BindingDiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.BindMethodBody(MethodSymbol method, TypeCompilationState compilationState, BindingDiagnosticBag diagnostics, Boolean includeInitializersInBody, BoundNode initializersBody, Boolean reportNullableDiagnostics, ImportChain& importChain, Boolean& originalBodyNested, Boolean& prependedDefaultValueTypeConstructorInitializer, InitialState& forSemanticModel)
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileMethod(MethodSymbol methodSymbol, Int32 methodOrdinal, ProcessedFieldInitializers& processedInitializers, SynthesizedSubmissionFields previousSubmissionFields, TypeCompilationState compilationState)
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileNamedType(NamedTypeSymbol containingType)
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.<>c__DisplayClass25_0.<CompileNamedTypeAsync>b__0()
   at Roslyn.Utilities.UICultureUtilities.<>c__DisplayClass5_0.<WithCurrentUICulture>b__0()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.WaitForWorkers()
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileMethodBodies(CSharpCompilation compilation, PEModuleBuilder moduleBeingBuiltOpt, Boolean emittingPdb, Boolean hasDeclarationErrors, Boolean emitMethodBodies, BindingDiagnosticBag diagnostics, Predicate`1 filterOpt, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.CompileMethods(CommonPEModuleBuilder moduleBuilder, Boolean emittingPdb, DiagnosticBag diagnostics, Predicate`1 filterOpt, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.Compilation.Emit(Stream peStream, Stream metadataPEStream, Stream pdbStream, Stream xmlDocumentationStream, Stream win32Resources, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, Stream sourceLinkStream, IEnumerable`1 embeddedTexts, RebuildData rebuildData, CompilationTestData testData, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.Compilation.Emit(Stream peStream, Stream pdbStream, Stream xmlDocumentationStream, Stream win32Resources, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, Stream sourceLinkStream, IEnumerable`1 embeddedTexts, Stream metadataPEStream, RebuildData rebuildData, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.Compilation.Emit(Stream peStream, Stream pdbStream, Stream xmlDocumentationStream, Stream win32Resources, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, Stream sourceLinkStream, IEnumerable`1 embeddedTexts, Stream metadataPEStream, CancellationToken cancellationToken)
   at Stryker.Core.Compiling.CsharpCompilingProcess.TryCompilation(Stream ms, Stream symbolStream, CSharpCompilation compilation, EmitResult previousEmitResult, Boolean lastAttempt, Int32 retryCount) in C:\Dev\Repos\stryker-net\src\Stryker.Core\Stryker.Core\Compiling\CsharpCompilingProcess.cs:line 172
   at Stryker.Core.Compiling.CsharpCompilingProcess.Compile(IEnumerable`1 syntaxTrees, Stream ilStream, Stream symbolStream) in C:\Dev\Repos\stryker-net\src\Stryker.Core\Stryker.Core\Compiling\CsharpCompilingProcess.cs:line 63
   at Stryker.Core.MutationTest.CsharpMutationProcess.CompileMutations(MutationTestInput input, CsharpCompilingProcess compilingProcess) in C:\Dev\Repos\stryker-net\src\Stryker.Core\Stryker.Core\MutationTest\CsharpMutationProcess.cs:line 89
   at Stryker.Core.MutationTest.CsharpMutationProcess.Mutate(MutationTestInput input) in C:\Dev\Repos\stryker-net\src\Stryker.Core\Stryker.Core\MutationTest\CsharpMutationProcess.cs:line 79
   at Stryker.Core.MutationTest.MutationTestProcess.Mutate() in C:\Dev\Repos\stryker-net\src\Stryker.Core\Stryker.Core\MutationTest\MutationTestProcess.cs:line 88
   at Stryker.Core.Initialisation.ProjectMutator.MutateProject(StrykerOptions options, MutationTestInput input, IReporter reporters) in C:\Dev\Repos\stryker-net\src\Stryker.Core\Stryker.Core\Initialisation\ProjectMutator.cs:line 38
   at Stryker.Core.Initialisation.ProjectOrchestrator.MutateProjects(StrykerOptions options, IReporter reporters, ITestRunner runner)+MoveNext() in C:\Dev\Repos\stryker-net\src\Stryker.Core\Stryker.Core\Initialisation\ProjectOrchestrator.cs:line 68
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Stryker.Core.StrykerRunner.RunMutationTest(IStrykerInputs inputs, ILoggerFactory loggerFactory, IProjectOrchestrator projectOrchestrator) in C:\Dev\Repos\stryker-net\src\Stryker.Core\Stryker.Core\StrykerRunner.cs:line 63
   at Stryker.CLI.StrykerCli.RunStryker(IStrykerInputs inputs) in C:\Dev\Repos\stryker-net\src\Stryker.CLI\Stryker.CLI\StrykerCLI.cs:line 102
   at Stryker.CLI.StrykerCli.<>c__DisplayClass11_0.<Run>b__0() in C:\Dev\Repos\stryker-net\src\Stryker.CLI\Stryker.CLI\StrykerCLI.cs:line 74
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>c__DisplayClass143_0.<OnExecute>b__0(CancellationToken _)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Stryker.CLI.StrykerCli.Run(String[] args) in C:\Dev\Repos\stryker-net\src\Stryker.CLI\Stryker.CLI\StrykerCLI.cs:line 80
   at Stryker.CLI.Program.Main(String[] args) in C:\Dev\Repos\stryker-net\src\Stryker.CLI\Stryker.CLI\Program.cs:line 14

Logs log-20240315.txt

Desktop (please complete the following information):

• OS: Windows
• Type of project: Stryker latest master branch
• Stryker Version: 4.0.0

Additional context I reverted some commits to locate that this behaviour was added during the refactor of the mutation orchestration by @dupdob

[dupdob]

I reproduce, looking into it. Crashing the compiler is pretty rare...

[dupdob]

ok, this is the mutated version of GitInfoProvider that triggers this. It looks like there are ~80 mutations there. Finding the problematic one will be fun

[richardwerkman]

Ah man, good luck analyzing this :) Thanks for looking into this so quickly!

[dupdob]

look what Copilot proposed me while working on a recovery strategy: _logger.LogError("Failed to compile due to a NullReferenceException. This is a known issue in Roslyn. Please report the issue to the Roslyn team."); So far, I did not spot anything surprising while reading the mutated code. But I am almost done having a work around strategy for his kind of issues

• [x] identify the problematic file(s) and logs its content
• [x] remove all mutations from this file and flag them as compilation error
• [ ] restore the normal Stryker flows
• [x] spot what syntax construct triggered the error
• [ ] fix Stryker accordingly
• [ ] alt. devise a better workaround if this is a genuine Roslyn bug

[dupdob]

update: it looks like there are 5-6 files that raises this problem. I am not able yet to restore normal flow. This probably due to some stupid bug on my side as restoring the concerned files to their original version should remove the exception. re update: stupid error found, working on a proper fix (this was due to rollback logic reusing non fixed syntax trees)

[dupdob]

minor update: of course, this issue appears also in single project mode. I fixed source file compilation issues, but I still have problem with project resources.

[richardwerkman]

Huh, this seems like a big issue. It's a shame our integration test didn't catch this...

[dupdob]

I still have problems with resources management. It is likely to be a side effect of the compiler exception which triggers some cleanup. I have identified one problematic construct, so I should be able to reproduce and fix it: _testGuids?.Count ?? 0; These are not properly mutated. I will not share the mutated as it contains 4 mutations. Clearly the conditional operator was not properly mutated (this is the last mutation) ...(StrykerabBot37wrmyO0As.MutantControl.IsActive(3055)?.Sum:.Count)

[dupdob]

update: the root cause is a regression with ConditionalAccessExpressions (?.). I am trying to find a way to deal with these in some central fashion, and not by adding ifs everywhere. My last refactoring went too far; early design was too clunky, the current one is too streamlined. I may have a better design in mind. We will see about that.

I have been able to successfully compile 2 of the problematic files, the remaining three contains various constructions with conditional access expressions that force me to think about a better design.

Regarding restoring proper workflow in case of Roslyn exception, I have an almost working version, but I still have issues with resources: streams are finalized when the exception is thrown. I will see what can be done about it, but my focus is on the conditional access expression stuff.

Edited to correct usage of ConditionalExpressions (?:) instead of the proper ConditionalAccessExpression (?.) Stupid me

[dupdob]

update: the design I had in mind works. I had to add several orchestrators, which I do not like. But, I finally got my head around chained MemberAccesses (and variants). Still have to streamline the design a bit. But I can open a PR now if the fix is urgent

[mu88]

Nothing except going to the hospital with a missing leg is really urgent :) just let us know what you prefer: shall we better stick to the old version so that you have more time to find a solution that fits you better? Or wait for a temporary fix?
Because in our CI pipeline, we're pulling and installing the latest version of Stryker (which fails at the moment) and now I have to decide whether to live with the failing builds for a couple of days or go back to the previous version :)

[dupdob]

I will see today how far I am for a good solution and decide afterward. One thing is sure: there is no quick fix, it needs to add a few orchestrators and maybe add features to MutationContext. So if I think this is too far away, I will open a PR with an interim solution. Here are the things I want to fix:

• [x] restore the normal Stryker flows (not looking into the resource problem yet
• [x] streamline the current fix: the new orchestrators may be reduced to a smaller code base via some shared base class, for example
• [x] increase mutation agility: as of now, Stryker can only mutate whole member access chains, such as x.y.z().t.u, at once, while it would be smarter to mutate only the beginning (which the master version does, sometimes).

The problem with member access chains is that they are described right from left, which may force some double scanning...

[richardwerkman]

@mu88 I was wondering if more people were seeing this behavior. We could see if we can revert the commit that causes this for now. And release the refactor again later (when a fix is ready).

[dupdob]

cool. I have improved design and the recovery mechanism now works properly. Still need to:

• [x] update technical doc
• [ ] try to add tests for recovery mechanism (not sure I can). Update: it requires mocking CSharpCompilation, which requires wrapping it (not done today). I will not do it for this fix (not sure it is worth the hassle and delay)

[mu88]

@mu88 I was wondering if more people were seeing this behavior. We could see if we can revert the commit that causes this for now. And release the refactor again later (when a fix is ready).

@richardwerkman sry for not jumping in earlier ^^ I saw the issue popping up last week on our infrastructure, however, I was triggering figuring out what's wrong on our side because I didn't blame Stryker :)

[dupdob]

the PR is ready for review. As said earlier, I will not attempt to reach expected coverage level as it requires wrapping C# compiler