assemblyloading_assemblylocationnotadded always failed in Linux

[runfoapp[bot]]

Runfo Tracking Issue: assemblyloading_assemblylocationnotadded always failed in Linux	Build	Definition	Kind	Run Name

Build Result Summary	Day Hit Count	Week Hit Count	Month Hit Count
0	0	0

[Cosifne]

Note: This is merged into main https://github.com/dotnet/roslyn/pull/66492 But when run later, block other PR like https://github.com/dotnet/roslyn/pull/66613 I tried rerun many times, no success.

The stack trace is:

System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.

Stack trace
   at Microsoft.CodeAnalysis.SeparatedSyntaxList`1.System.Collections.Generic.IEnumerable<TNode>.GetEnumerator()
   at System.Collections.Immutable.ImmutableArray`1.Builder.AddRange(IEnumerable`1 items)
   at Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder`1.AddRange(IEnumerable`1 items) in /_/src/Dependencies/PooledObjects/ArrayBuilder.cs:line 555
   at Microsoft.CodeAnalysis.CSharp.Symbol.GetAttributesToBind(OneOrMany`1 attributeDeclarationSyntaxLists, AttributeLocation symbolPart, BindingDiagnosticBag diagnostics, CSharpCompilation compilation, Func`2 attributeMatchesOpt, Binder rootBinderOpt, ImmutableArray`1& binders) in /_/src/Compilers/CSharp/Portable/Symbols/Symbol_Attributes.cs:line 603
   at Microsoft.CodeAnalysis.CSharp.Symbol.LoadAndValidateAttributes(OneOrMany`1 attributesSyntaxLists, CustomAttributesBag`1& lazyCustomAttributesBag, AttributeLocation symbolPart, Boolean earlyDecodingOnly, Binder binderOpt, Func`2 attributeMatchesOpt, Action`1 beforeAttributePartBound, Action`1 afterAttributePartBound) in /_/src/Compilers/CSharp/Portable/Symbols/Symbol_Attributes.cs:line 304
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceAssemblySymbol.EnsureAttributesAreBound() in /_/src/Compilers/CSharp/Portable/Symbols/Source/SourceAssemblySymbol.cs:line 1559
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceAssemblySymbol.GetSourceAttributesBag() in /_/src/Compilers/CSharp/Portable/Symbols/Source/SourceAssemblySymbol.cs:line 1574
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceAssemblySymbol.GetSourceDecodedWellKnownAttributeData() in /_/src/Compilers/CSharp/Portable/Symbols/Source/SourceAssemblySymbol.cs:line 1640
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceAssemblySymbol.get_AssemblyDelaySignAttributeSetting() in /_/src/Compilers/CSharp/Portable/Symbols/Source/SourceAssemblySymbol.cs:line 323
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceAssemblySymbol.get_IsDelaySigned() in /_/src/Compilers/CSharp/Portable/Symbols/Source/SourceAssemblySymbol.cs:line 881
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.get_IsDelaySigned() in /_/src/Compilers/CSharp/Portable/Compilation/CSharpCompilation.cs:line 3116
   at Microsoft.CodeAnalysis.Compilation.get_HasStrongName() in /_/src/Compilers/Core/Portable/Compilation/Compilation.cs:line 2388
   at Microsoft.CodeAnalysis.Compilation.ConstructModuleSerializationProperties(EmitOptions emitOptions, String targetRuntimeVersion, Guid moduleVersionId) in /_/src/Compilers/Core/Portable/Compilation/Compilation.cs:line 2254
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.CreateModuleBuilder(EmitOptions emitOptions, IMethodSymbol debugEntryPoint, Stream sourceLinkStream, IEnumerable`1 embeddedTexts, IEnumerable`1 manifestResources, CompilationTestData testData, DiagnosticBag diagnostics, CancellationToken cancellationToken) in /_/src/Compilers/CSharp/Portable/Compilation/CSharpCompilation.cs:line 3143
   at Microsoft.CodeAnalysis.Compilation.CheckOptionsAndCreateModuleBuilder(DiagnosticBag diagnostics, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, Stream sourceLinkStream, IEnumerable`1 embeddedTexts, CompilationTestData testData, CancellationToken cancellationToken) in /_/src/Compilers/Core/Portable/Compilation/Compilation.cs:line 3143
   at Microsoft.CodeAnalysis.Compilation.Emit(Stream peStream, Stream metadataPEStream, Stream pdbStream, Stream xmlDocumentationStream, Stream win32Resources, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, Stream sourceLinkStream, IEnumerable`1 embeddedTexts, RebuildData rebuildData, CompilationTestData testData, CancellationToken cancellationToken) in /_/src/Compilers/Core/Portable/Compilation/Compilation.cs:line 2902

Another example run https://dev.azure.com/dnceng-public/public/_build/results?buildId=154270&view=results

[Cosifne]

cc @jaredpar

[jaredpar]

Context on this failure.

The code that introduced this failure was merged in #66492. During the review of that change we saw one test consistently failing on Linux and followed up with the runtime team in https://github.com/dotnet/runtime/issues/81108. Once we confirmed there was a bug over there we disabled that test on Linux, continued working on the PR and didn't see any future issues. The PR was merged with several clean green runs.

Post merge we saw a different test fail on Linux. That was disabled and then another one started failing. At that point @Cosifne and I decided to just stop these tests running on Linux as fast as possible to get the builds moving again (https://github.com/dotnet/roslyn/commit/c3afa4eb9b2ce63e7e93eeaa602066d43f4cb343).

It's unclear exactly what happened post merge to cause this to spread to most Linux runs. My suspicion is test partitioning changed such that it put more pressure on the underlying runtime issue resulting in wider spread failures. No evidence to back this up though, just a suspicion.

The PR #66612 reduces the compilation calls which appear to be putting pressure on the runtime by ~95%. Hopefully that will allow us to re-enable the Linux runs.