search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.93k stars 4.64k forks source link

Seg fault in Linq.Expression tests #105706

Closed ericstj closed 1 month ago

ericstj commented 1 month ago

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=759698 Build error leg or test failing: System.Linq.Expressions.Tests.WorkItemExecution Pull request: https://github.com/dotnet/runtime/pull/105636

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": ["SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer", "System.Linq.Expressions.Interpreter.FuncCallInstruction"],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Log: https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-105636-merge-95523dd4fead49c19e/System.Linq.Expressions.Tests/1/console.673b37a5.log?helixlogtype=result Dump: https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-105636-merge-95523dd4fead49c19e/System.Linq.Expressions.Tests/1/coredump.20.dmp?helixlogtype=result

Relevant portion of crash analysis, some symbols missing cc @hoyosjs 

Thread Id: 0x22
      Child SP               IP Call Site
 0x7294dbb1f260 0x72d5778fdef9 0x72d5778fdef9
 0x7294dbb1f310 0x72d576e924bb System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.FuncCallInstruction`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]]..ctor(System.Reflection.MethodInfo)
 0x7294dbb1f3a0 0x72d5f4cc3be4 0x72d5f4cc3be4
 0x7294dbb1f3c0 0x72d5f4b01155 libcoreclr.so!?? at ??:0:0
 0x7294dbb1f400 0x72d5f4bab122 libcoreclr.so!?? at ??:0:0
 0x7294dbb1f7a0 0x72d5779304a1 System.Private.CoreLib.dll!System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(System.Object, System.Span`1<System.Object>, System.Reflection.BindingFlags)
 0x7294dbb1f820 0x72d57791a176 System.Private.CoreLib.dll!System.Reflection.MethodBaseInvoker.InvokeWithOneArg(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
 0x7294dbb1f8c0 0x72d5777fba77 System.Private.CoreLib.dll!System.Reflection.RuntimeConstructorInfo.Invoke(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
 0x7294dbb1f920 0x72d576af0676 System.Private.CoreLib.dll!System.RuntimeType.CreateInstanceImpl(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
 0x7294dbb1f9f0 0x72d576acaac7 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.CallInstruction.SlowCreate(System.Reflection.MethodInfo, System.Reflection.ParameterInfo[])
 0x7294dbb1faf0 0x72d576ac67c2 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.CallInstruction.FastCreate(System.Reflection.MethodInfo, System.Reflection.ParameterInfo[])
 0x7294dbb1fc20 0x72d576ac60a0 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.CallInstruction.Create(System.Reflection.MethodInfo, System.Reflection.ParameterInfo[])
 0x7294dbb1fda0 0x72d576ac5495 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.InstructionList.EmitCall(System.Reflection.MethodInfo, System.Reflection.ParameterInfo[])
 0x7294dbb1fde0 0x72d576abf14d System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.LightCompiler.CompileMethodCallExpression(System.Linq.Expressions.Expression, System.Reflection.MethodInfo, System.Linq.Expressions.IArgumentProvider)
 0x7294dbb1ff50 0x72d577210910 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.LightCompiler.CompileMethodCallExpression(System.Linq.Expressions.Expression)
 0x7294dbb1ff90 0x72d576abd924 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.LightCompiler.CompileNoLabelPush(System.Linq.Expressions.Expression)
 0x7294dbb201c0 0x72d576abc619 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.LightCompiler.Compile(System.Linq.Expressions.Expression)
 0x7294dbb201f0 0x72d576abc349 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.LightCompiler.CompileTop(System.Linq.Expressions.LambdaExpression)
 0x7294dbb20320 0x72d576e86be9 System.Linq.Expressions.dll!System.Linq.Expressions.LambdaExpression.Compile(Boolean)
 0x7294dbb20370 0x72d578fef633 System.Linq.Expressions.Tests.dll!System.Linq.Expressions.Tests.CallTests.Call_NoParameters(System.Linq.Expressions.Expression, System.Reflection.MethodInfo, System.Object, Boolean)
 0x7294dbb20400 0x72d577aab5b2 System.Private.CoreLib.dll!DynamicClass.InvokeStub_CallTests.Call_NoParameters(System.Object, System.Span`1<System.Object>)

Could be reflection, or codegen. cc @steveharter @AndyAyersMS in case they see anything.

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=759698 Error message validated: [SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer System.Linq.Expressions.Interpreter.FuncCallInstruction] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 7/30/2024 4:08:57 PM UTC

Report

Build Definition Test Pull Request
771458 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution
771358 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#106083
770443 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#106167
770388 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#106165
770213 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#106163
768825 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105903
768664 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#106078
768349 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#99596
768237 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution
767931 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#106053
767917 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105841
767251 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#106015
767206 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105941
766489 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution
766122 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105941
765937 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105866
765575 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105928
765321 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105868
764956 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105909
764852 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution
764572 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105846
764162 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105875
763929 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105841
763102 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105666
763231 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#101963
763107 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105826
762507 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105050
761806 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#104562
761539 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105692
761433 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105749
760893 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105300
760243 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105471
760176 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105689
760299 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105610
760097 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105680
759698 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105636
759000 dotnet/runtime System.Linq.Expressions.Tests.WorkItemExecution dotnet/runtime#105308

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
4 20 37
dotnet-policy-service[bot] commented 1 month ago

Tagging subscribers to this area: @cston See info in area-owners.md if you want to be subscribed.

cston commented 1 month ago

See also https://github.com/dotnet/runtime/issues/105704.

AndyAyersMS commented 1 month ago

Let me take a look; this seems to be happing frequently.

AndyAyersMS commented 1 month ago

Based on the windows x64 crash in https://dev.azure.com/dnceng-public/public/_build/results?buildId=764230&view=ms.vss-test-web.build-test-results-tab&runId=19436920&resultId=218614&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab

There is a delegate that blows up the process when invoked. The method invoking the delegate is minopts.

0:010> !DumpObj /d 00000136ce117488
Name:        System.Func`2[[System.Linq.Expressions.Tests.IncDecAssignTests+TestPropertyClass`1[[System.Int32, System.Private.CoreLib]], System.Linq.Expressions.Tests],[System.Int32, System.Private.CoreLib]]
MethodTable: 00007fff9bd6aaa0
EEClass:     00007fff9a59b7e8
Tracked Type: false
Size:        64(0x40) bytes
File:        C:\h\w\B8A309B6\p\shared\Microsoft.NETCore.App\9.0.0\System.Private.CoreLib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007fff998d4530  400021c        8        System.Object  0 instance 00000136ce117488 _target
00007fff998d4530  400021d       10        System.Object  0 instance 00000136ce116978 _methodBase
00007fff99a15170  400021e       18        System.IntPtr  1 instance 00007FFF9AD5F010 _methodPtr
00007fff99a15170  400021f       20        System.IntPtr  1 instance 00007FFF9BD4A0E8 _methodPtrAux
00007fff998d4530  40002c3       28        System.Object  0 instance 0000000000000000 _invocationList
00007fff99a15170  40002c4       30        System.IntPtr  1 instance 0000000000000000 _invocationCount

Here the methodPtr is an invalid address. The methodPtrAux field is an indirection cell for

0:010> !ip2md 00007fff`9ba11e90 
MethodDesc:   00007fff9bd6a100
Method Name:          System.Linq.Expressions.Tests.IncDecAssignTests+TestPropertyClass`1[[System.Int32, System.Private.CoreLib]].get_TestInstance()
Class:                00007fff9bd6a148
MethodTable:          00007fff9bd6a148
mdToken:              0000000006004518
Module:               00007fff9a2e25a0
IsJitted:             yes
Current CodeAddr:     00007fff9ba11e90
Version History:
  ILCodeVersion:      0000000000000000
  ReJIT ID:           0
  IL Addr:            00000177600f5b5b
     CodeAddr:           00007fff9ba11e90  (MinOptJitted)
     NativeCodeVersion:  0000000000000000

Locally (using CI assets) I was able to get 5 crashes in 200 runs, so I may be able to catch this live in the debugger.

Aside from the AV there were also crashes like

Fatal error. Internal CLR error. (0x80131506)
   at System.Delegate.<BindToMethodInfo>g____PInvoke|21_0(System.Runtime.CompilerServices.ObjectHandleOnStack, System.Runtime.CompilerServices.ObjectHandleOnStack, System.RuntimeMethodHandleInternal, System.Runtime.CompilerServices.QCallTypeHandle, System.DelegateBindingFlags)
   at System.Delegate.CreateDelegateInternal(System.RuntimeType, System.Reflection.RuntimeMethodInfo, System.Object, System.DelegateBindingFlags)
   at System.Reflection.RuntimeMethodInfo.CreateDelegateInternal(System.Type, System.Object, System.DelegateBindingFlags)
   at System.Linq.Expressions.Interpreter.FuncCallInstruction`2[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Int32, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor(System.Reflection.MethodInfo)
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, Void**, System.Signature, Boolean)
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(System.Object, System.Span`1<System.Object>, System.Reflection.BindingFlags)
   at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)

Based on the above my guess is that this is an issue in the runtime with stub management?

cc @mangod9

mangod9 commented 1 month ago

Don't believe there have been any recent changes I am aware of which might affect this. We can take a look though. Also adding @janvorli @VSadov if it rings a bell?

AndyAyersMS commented 1 month ago

Can't quite figure out how to script this under the debugger, because some of the tests intentionally divide by zero.

steveharter commented 1 month ago

Adding @AaronRobinsonMSFT ? - some delegate changes were made in https://github.com/dotnet/runtime/pull/105584 on Jul 27 which was a couple days before the first report on July 29.

mangod9 commented 1 month ago

this looks similar to https://github.com/dotnet/runtime/pull/106072#issuecomment-2276356612, and a possible root cause PR has been reverted.

AaronRobinsonMSFT commented 1 month ago

I think this was fixed with https://github.com/dotnet/runtime/pull/106072#issuecomment-2278438726

/cc @jkotas