stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.76k stars 175 forks source link

Linq mutator creates invalid syntax when conditional access expressions encountered #704

Closed rouke-broersma closed 4 years ago

rouke-broersma commented 4 years ago

Describe the bug We seem to be hitting https://github.com/dotnet/roslyn/issues/25262 It looks like we are generating valid syntax that is somehow not supported by the roslyn compiler, causing it to crash in a nasty way.

Nailed down to issue in linq mutator so far.

Desktop (please complete the following information):

[21:03:32 ERR] An error occurred during the mutation test run
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.CodeAnalysis.CSharp.SyntaxFactory.FindConditionalAccessNodeForBinding(CSharpSyntaxNode node) in /_/src/Compilers/CSharp/Portable/Syntax/SyntaxFactory.cs:line 2224
   at Microsoft.CodeAnalysis.CSharp.Binder.GetReceiverForConditionalBinding(ExpressionSyntax binding, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 8059
   at Microsoft.CodeAnalysis.CSharp.Binder.BindMemberBindingExpression(MemberBindingExpressionSyntax node, Boolean invoked, Boolean indexed, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 8028
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 435
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 329
   at Microsoft.CodeAnalysis.CSharp.Binder.BindLeftOfPotentialColorColorMemberAccess(ExpressionSyntax left, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 5531
   at Microsoft.CodeAnalysis.CSharp.Binder.BindMemberAccess(MemberAccessExpressionSyntax node, Boolean invoked, Boolean indexed, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 5430
   at Microsoft.CodeAnalysis.CSharp.Binder.BindInvocationExpression(InvocationExpressionSyntax node, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Invocation.cs:line 160
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 384
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 329
   at Microsoft.CodeAnalysis.CSharp.Binder.BindConditionalOperator(ConditionalExpressionSyntax node, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Operators.cs:line 3731
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 473
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 329
   at Microsoft.CodeAnalysis.CSharp.Binder.BindParenthesizedExpression(ExpressionSyntax innerExpression, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 1109
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 492
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 329
   at Microsoft.CodeAnalysis.CSharp.Binder.BindConditionalAccessExpression(ConditionalAccessExpressionSyntax node, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 7920
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 432
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 329
   at Microsoft.CodeAnalysis.CSharp.Binder.BindSimpleBinaryOperator(BinaryExpressionSyntax node, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Operators.cs:line 427
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 425
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 329
   at Microsoft.CodeAnalysis.CSharp.Binder.BindConditionalOperator(ConditionalExpressionSyntax node, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Operators.cs:line 3732
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 473
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 329
   at Microsoft.CodeAnalysis.CSharp.Binder.BindParenthesizedExpression(ExpressionSyntax innerExpression, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 1109
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpressionInternal(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 492
   at Microsoft.CodeAnalysis.CSharp.Binder.BindExpression(ExpressionSyntax node, DiagnosticBag diagnostics, Boolean invoked, Boolean indexed) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs:line 329
   at Microsoft.CodeAnalysis.CSharp.Binder.BindBooleanExpression(ExpressionSyntax node, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Statements.cs:line 2207
   at Microsoft.CodeAnalysis.CSharp.Binder.BindIfStatement(IfStatementSyntax node, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Statements.cs:line 2163
   at Microsoft.CodeAnalysis.CSharp.Binder.BindStatement(StatementSyntax node, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Statements.cs:line 55
   at Microsoft.CodeAnalysis.CSharp.Binder.BindBlockParts(BlockSyntax node, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Statements.cs:line 1660
   at Microsoft.CodeAnalysis.CSharp.Binder.BindStatement(StatementSyntax node, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Statements.cs:line 43
   at Microsoft.CodeAnalysis.CSharp.Binder.BindMethodBody(CSharpSyntaxNode declaration, BlockSyntax blockBody, ArrowExpressionClauseSyntax expressionBody, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Statements.cs:line 3170
   at Microsoft.CodeAnalysis.CSharp.Binder.BindMethodBody(CSharpSyntaxNode syntax, DiagnosticBag diagnostics) in /_/src/Compilers/CSharp/Portable/Binder/Binder_Statements.cs:line 3118
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.BindMethodBody(MethodSymbol method, TypeCompilationState compilationState, DiagnosticBag diagnostics, ImportChain& importChain, Boolean& originalBodyNested, ValueTuple`4& forSemanticModel) in /_/src/Compilers/CSharp/Portable/Compiler/MethodCompiler.cs:line 1653
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileMethod(MethodSymbol methodSymbol, Int32 methodOrdinal, ProcessedFieldInitializers& processedInitializers, SynthesizedSubmissionFields previousSubmissionFields, TypeCompilationState compilationState) in /_/src/Compilers/CSharp/Portable/Compiler/MethodCompiler.cs:line 958
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileNamedType(NamedTypeSymbol containingType) in /_/src/Compilers/CSharp/Portable/Compiler/MethodCompiler.cs:line 507
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.<>c__DisplayClass22_0.<CompileNamedTypeAsTask>b__0() in /_/src/Compilers/CSharp/Portable/Compiler/MethodCompiler.cs:line 397
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.WaitForWorkers() in /_/src/Compilers/CSharp/Portable/Compiler/MethodCompiler.cs:line 310
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileMethodBodies(CSharpCompilation compilation, PEModuleBuilder moduleBeingBuiltOpt, Boolean emittingPdb, Boolean emitTestCoverageData, Boolean hasDeclarationErrors, DiagnosticBag diagnostics, Predicate`1 filterOpt, CancellationToken cancellationToken) in /_/src/Compilers/CSharp/Portable/Compiler/MethodCompiler.cs:line 156
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.CompileMethods(CommonPEModuleBuilder moduleBuilder, Boolean emittingPdb, Boolean emitMetadataOnly, Boolean emitTestCoverageData, DiagnosticBag diagnostics, Predicate`1 filterOpt, CancellationToken cancellationToken) in /_/src/Compilers/CSharp/Portable/Compilation/CSharpCompilation.cs:line 2723
   at Microsoft.CodeAnalysis.Compilation.Emit(Stream peStream, Stream metadataPEStream, Stream pdbStream, Stream xmlDocumentationStream, Stream win32Resources, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, Stream sourceLinkStream, IEnumerable`1 embeddedTexts, CompilationTestData testData, CancellationToken cancellationToken) in /_/src/Compilers/Core/Portable/Compilation/Compilation.cs:line 2404
   at Microsoft.CodeAnalysis.Compilation.Emit(Stream peStream, Stream pdbStream, Stream xmlDocumentationStream, Stream win32Resources, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, Stream sourceLinkStream, IEnumerable`1 embeddedTexts, Stream metadataPEStream, CancellationToken cancellationToken) in /_/src/Compilers/Core/Portable/Compilation/Compilation.cs:line 2352
   at Stryker.Core.Compiling.CompilingProcess.TryCompilation(MemoryStream ms, CSharpCompilation compilation, EmitResult previousEmitResult, Boolean devMode, Int32 retryCount) in C:\git\stryker-net\src\Stryker.Core\Stryker.Core\Compiling\CompilingProcess.cs:line 122
   at Stryker.Core.Compiling.CompilingProcess.Compile(IEnumerable`1 syntaxTrees, MemoryStream ms, Boolean devMode) in C:\git\stryker-net\src\Stryker.Core\Stryker.Core\Compiling\CompilingProcess.cs:line 70
   at Stryker.Core.MutationTest.MutationTestProcess.Mutate() in C:\git\stryker-net\src\Stryker.Core\Stryker.Core\MutationTest\MutationTestProcess.cs:line 113
   at Stryker.Core.StrykerRunner.RunMutationTest(StrykerOptions options) in C:\git\stryker-net\src\Stryker.Core\Stryker.Core\StrykerRunner.cs:line 80
dupdob commented 4 years ago

Any chance you can share the code that triggers this? Reading the issue you are referring to, there is a chance that the generated syntax is actually invalid, triggering an internal error in Roslyn. Assuming I read the issue properly, of course.

rouke-broersma commented 4 years ago

@dupdob The latest comment on that issue says the following:

From discussion with @gafter the compiler should be resilient to syntax that was (or could be) produced by the parser, which is the case here. There should be no exception at all.

So some valid syntax (according to the parser) is not valid syntax in the compiler is how I read it. I'm trying to narrow down which part of the code triggers it, but since the compiler has an NPE I cannot determine from the exception where the invalid syntax is. I'm investigating manually by excluding more and more files from the mutation run.

dupdob commented 4 years ago

I stand corrected. I (wrongly) understood the roslyn issue as 'syntactly correct' code that could not compile. It can be the case when referencing non existing type or members for example. In case that helps, the issue is linked to conditional access, ie the ?. operator.

rouke-broersma commented 4 years ago

I'm closing to finding the source code :p

image

dupdob commented 4 years ago

I suspected the Linq mutator because it is the only mutator changing member names

Envoyé de mon iPhone

Le 4 sept. 2019 à 09:17, Rouke Broersma notifications@github.com a écrit :

I'm closing to finding the source code :p

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

rouke-broersma commented 4 years ago

Offending source code:

        public Team GetTeam(string name)
        {
            var team = _teamRepository.GetTeam(name);
            if (team?.Features.Any(f => f.Name == _apiOptions.ManageTeamsFeatureName) == true)
            {
                return null;
            }
            else
            {
                return team;
            }
        }
rouke-broersma commented 4 years ago

Mutates to this abomination with only linq mutator turned on:

public Team GetTeam(string name)
{
    var team = _teamRepository.GetTeam(name);
    if (team?(StrykerjDVloRFwLAEiZZO.MutantControl.IsActive(16)?.Features.All(f => f.Name == _apiOptions.ManageTeamsFeatureName) :.Features.Any(f => f.Name == _apiOptions.ManageTeamsFeatureName) )== true)
    {
        return null;
    }
    else
    {
        return team;
    }
}

And this with all mutators turned on:

public Team GetTeam(string name)
{
    var team = _teamRepository.GetTeam(name);
    if ((StrykernXatqrr71KMnocx.MutantControl.IsActive(386)?team?.Features.Any(f => f.Name == _apiOptions.ManageTeamsFeatureName) != true:team?(StrykernXatqrr71KMnocx.MutantControl.IsActive(384)?.Features.All(f => f.Name == _apiOptions.ManageTeamsFeatureName) :.Features.Any(f => (StrykernXatqrr71KMnocx.MutantControl.IsActive(383)?f.Name != _apiOptions.ManageTeamsFeatureName:f.Name == _apiOptions.ManageTeamsFeatureName)) )== (StrykernXatqrr71KMnocx.MutantControl.IsActive(385)?false:true)))
    {
        return null;
    }
    else
    {
        return team;
    }
}
dupdob commented 4 years ago

My eyes! My eyes!

An abomination indeed!! Issue is obvious, cause is clear, fix will be tricky. The problem is that the conditional operator is not set on the proper expression. It can’t happen within a chain of member invocation, as ‘.’ is not an operator in C#.

dupdob commented 4 years ago

On a second though, the fix may not be that difficult; we should change the LinqMutator logic so that:

dupdob commented 4 years ago

I can confirm my previous analysis: the conditional operator presence results in a significantly different syntax tree, leading to the observed results.

richardwerkman commented 4 years ago

@dupdob will you try to submit a pull request? If so we can focus on some other issues 👍

dupdob commented 4 years ago

Ok. I take this issue

dupdob commented 4 years ago

PR #706 should fix this issue. Please verify with your code for confirmation.

rouke-broersma commented 4 years ago

Verified, thank you!

rouke-broersma commented 4 years ago

Fixed by #706