Closed kevingosse closed 3 years ago
Tagging subscribers to this area: @tommcdon See info in area-owners.md if you want to be subscribed.
Author: | kevingosse |
---|---|
Assignees: | - |
Labels: | `area-Diagnostics-coreclr`, `untriaged` |
Milestone: | - |
I am not familiar with this code path, but it may just be a bug on the runtime. There is no check for pExceptionRecord
being null here: https://github.com/dotnet/runtime/blob/f7de865465532de372f65b55e00a8ec2be1e5274/src/coreclr/vm/arm64/stubs.cpp#L1197
But UnwindManagedExceptionPass1
calls AdjustContextForVirtualStub
with null here: https://github.com/dotnet/runtime/blob/f7de865465532de372f65b55e00a8ec2be1e5274/src/coreclr/vm/exceptionhandling.cpp#L4579
I don't know off the top of my head why you would only see it when your profiler is attached, but it's suspicious that we check for null on other architectures: https://github.com/dotnet/runtime/blob/f7de865465532de372f65b55e00a8ec2be1e5274/src/coreclr/vm/amd64/excepamd64.cpp#L614-L617
I will have to do some research to convince myself that this is the right fix. @kevingosse, if you are feeling motivated it would be helpful to know if adding a check for null solves your problem.
I will have to do some research to convince myself that this is the right fix. @kevingosse, if you are feeling motivated it would be helpful to know if adding a check for null solves your problem.
I'll give it a try tomorrow. That would be my first time building the CLR on ARM64, but I suppose it shouldn't be much harder than x64.
It's largely the same except setting up the cross building environment. It's documented here: https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/cross-building.md, we have docker images with the environment preconfigured that makes it a lot easier.
I have more information on the issue. There are actually 2 segfaults. The first is at:
* thread #20, name = '.NET ThreadPool', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
frame #0: 0x0000ffff7d1fbee8
(lldb) clrstack
OS Thread Id: 0xbc2d (20)
Child SP IP Call Site
0000FFFF457F0F38 0000ffff7d1fbee8 [HelperMethodFrame: 0000ffff457f0f38]
0000FFFF457F1070 0000FFFF80D2AF6C System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
0000FFFF457F1090 0000FFFF80D2A230 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
0000FFFF457F1110 0000FFFF80D29A54 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
0000FFFF457F1150 0000FFFF80844360 System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(System.Threading.Tasks.Task)
0000FFFF457F1170 0000FFFF80844200 System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[[System.__Canon, System.Private.CoreLib]].GetResult()
0000FFFF457F11A0 0000FFFF829634B8 Elasticsearch.Net.RequestPipeline+<CallElasticsearchAsync>d__74`1[[System.__Canon, System.Private.CoreLib]].MoveNext() [C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs @ 467]
Then I resume execution from this point, it triggers another segfault, which is the one I reported:
* thread #19, name = '.NET ThreadPool', stop reason = signal SIGSEGV: invalid address (fault address: 0x10)
frame #0: 0x0000fffff72d0c3c libcoreclr.so`AdjustContextForVirtualStub(pExceptionRecord=0x0000000000000000, pContext=0x0000ffff45fecce0) at stubs.cpp:1173:40
Adding the null-check as you suggested fixes that second segfault. The first one still happens (I don't know if that's expected) but it does not crash the process anymore.
Do you have a coredump for the first segfault? Also, what does the native stack look like at the first segfault?
Do you have a coredump for the first segfault? Also, what does the native stack look like at the first segfault?
I can try capturing one tomorrow. There's no native stack (top frame is 0x0000ffff7d1fbee8
, which is the HelperMethodFrame). I'm not familiar with how those frame helpers work. For reference, where can I find the code that runs as part of the frame?
Frames (the coreclr data structure, they are confusingly named the same as stack frames) are what we use to track native code that is used by the runtime but needs to act like managed code. Each thread has a list of Frames that the StackWalker can use to determine if the code it is walking is one of our FCalls/Helpers/etc that plays by the same rules as managed code.
The HelperMethodFrame means that it is a jit helper. You should be able to look at the assembly of the helper by disassembling at the IP (0000ffff7d1fbee8) and you can inspect the HelperMethodFrame by looking at the address of the frame (0000ffff457f0f38). E.g. in lldb expr (HelperMethodFrame *)0000ffff457f0f38
should let you look at it. I'm typing this all from memory so beware of typos.
Just because it segfaults doesn't mean it's a bug though. You'd have to determine what the helper is and what it's supposed to be doing. NullReferenceExceptions in managed code are achieved by letting the native jitted code run and then if a segfault happens we look at the address of the segfault, and if it's in managed code we translate it to a managed NullReferenceException. The code that does that is what you added a NULL check to. So long story short, this could very well be normal operation.
Unfortunately I'm not sure how to capture a coredump for this segfault (since it doesn't crash the process). Saving a coredump from LLDB doesn't seem to be supported on this OS. I can't use gcore because the debugger is attached. I can capture one with GDB but the format is not supported by LLDB, and GDB seems to struggle to reconstruct the callstack.
In any case, I did more digging with LLDB. I see no evidence in the helper frame of what method is being called (m_FCallEntry
is NULL). The perf map says the current IP is mapped to GenerateDispatchStub<GenerateDispatchStub>
.
The disassembly of the stub is:
-> 0xffff7d21bd08: ldr x13, [x0]
0xffff7d21bd0c: adr x9, #0x1c
0xffff7d21bd10: ldp x10, x12, [x9]
0xffff7d21bd14: cmp x13, x10
0xffff7d21bd18: b.ne 0xffff7d21bd20
0xffff7d21bd1c: br x12
0xffff7d21bd20: ldr x9, #0x18
0xffff7d21bd24: br x9
It fails on the very first instruction (the value of x0 is 0x0). I'm going to assume this is this code: https://github.com/dotnet/runtime/blob/main/src/coreclr/vm/arm/stubs.cpp#L957 (I can't find a version for ARM64, does it mean it uses the version for ARM?) in which case the segfault is expected (according to the comment) just like you described.
Though it's in the middle of throwing a JSON serialization exception, it seems weird there would be a null reference there.
@kevingosse re: the first segfault, do you happen to have the call and data that triggers this? Might possibly be a bug in the json serializer (Utf8Json fork in 7.x) in the Elasticsearch .NET client that would be good for us to fix π In the interests of not wanting to derail this issue, an issue can be opened on https://github.com/elastic/elasticsearch-net/issues/new/choose
@kevingosse re: the first segfault, do you happen to have the call and data that triggers this? Might possibly be a bug in the json serializer (Utf8Json fork in 7.x) in the Elasticsearch .NET client that would be good for us to fix π In the interests of not wanting to derail this issue, an issue can be opened on https://github.com/elastic/elasticsearch-net/issues/new/choose
That's in the version 6 of the client, so still based on JSON.NET. In the end, the segfault happen in ExceptionDispatchInfo.Throw
so I doubt the Elasticsearch client is to blame, but if I pinpoint it to something in your code I'll make sure to report it π
From the runtime side I don't think that this is a bug. I don't look at codegen all that often so I could always be wrong, but I am pretty sure x0 is the object that is being dispatched, so if x0 is null that means someone is trying to call a virtual method on a null object.
Looking at ExceptionDispatchInfo.Throw it's a fairly simple method: https://github.com/dotnet/runtime/blob/bd7630dee87742594f1d8ec40cb5116c8b34528b/src/libraries/System.Private.CoreLib/src/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs#L52-L57
It seems likely that _exception is null here. How _exception is null is not obvious, since the ExceptionDispatchInfo would be returned by whatever Task is running, either through task.GetCancellationExceptionDispatchInfo() or task.GetExceptionDispatchInfos().
Are you able to capture it under a managed debugger so you can inspect the managed state?
I retrieved the ExceptionDispatchInfo instance with clrstack -a
, and everything looks fine to me:
(lldb) dumpobj 0x0000ffff48a6d198
Name: System.Runtime.ExceptionServices.ExceptionDispatchInfo
MethodTable: 0000ffff80356b78
EEClass: 0000ffff80362168
Size: 64(0x40) bytes
File: /home/ubuntu/.dotnet/shared/Microsoft.NETCore.App/5.0.4/System.Private.CoreLib.dll
Fields:
MT Field Offset Type VT Attr Value Name
0000ffff7e558710 4000e19 8 System.Exception 0 instance 0000ffff48a6ad40 _exception
0000ffff80356aa8 4000e1a 10 ...ion+DispatchState 1 instance 0000ffff48a6d1a8 _dispatchState
(lldb) dumpobj 0000ffff48a6ad40
Name: Nest.Json.JsonSerializationException
MethodTable: 0000ffff81ba73a0
EEClass: 0000ffff81bb1f48
Size: 128(0x80) bytes
File: /home/ubuntu/git/dd-trace-dotnet/test/test-applications/integrations/Samples.Elasticsearch/bin/Debug/net5.0/publish/Nest.dll
Fields:
MT Field Offset Type VT Attr Value Name
0000ffff7e583580 4000153 8 ...ection.MethodBase 0 instance 0000000000000000 _exceptionMethod
0000ffff7e554928 4000154 10 System.String 0 instance 0000ffff48a6a810 _message
0000ffff7e5a7ac8 4000155 18 ...tions.IDictionary 0 instance 0000000000000000 _data
0000ffff7e558710 4000156 20 System.Exception 0 instance 0000000000000000 _innerException
0000ffff7e554928 4000157 28 System.String 0 instance 0000000000000000 _helpURL
0000ffff7e7f39d8 4000158 30 System.Byte[] 0 instance 0000ffff48a6d860 _stackTrace
0000ffff7e7f39d8 4000159 38 System.Byte[] 0 instance 0000000000000000 _watsonBuckets
0000ffff7e554928 400015a 40 System.String 0 instance 0000000000000000 _stackTraceString
0000ffff7e554928 400015b 48 System.String 0 instance 0000000000000000 _remoteStackTraceString
0000ffff7e44ea48 400015c 50 System.Object[] 0 instance 0000000000000000 _dynamicMethods
0000ffff7e554928 400015d 58 System.String 0 instance 0000000000000000 _source
0000ffff7e523890 400015e 60 System.UIntPtr 1 instance 0000000000000000 _ipForWatsonBuckets
0000ffff7e5227f0 400015f 68 System.IntPtr 1 instance 0000000000000000 _xptrs
0000ffff7e4f9ba0 4000160 70 System.Int32 1 instance -532462766 _xcode
0000ffff7e4f9ba0 4000161 74 System.Int32 1 instance -2146233088 _HResult
I then checked the disassembly of ExceptionDispatchInfo.Throw
to try and find exactly where we stopped:
(lldb) clrstack
OS Thread Id: 0xd5d1 (20)
Child SP IP Call Site
0000FFFF44FF01F8 0000ffff7d1fbca8 [HelperMethodFrame: 0000ffff44ff01f8]
0000FFFF44FF0330 0000FFFF80E39A84 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
0000FFFF44FF0350 0000FFFF80E392D8 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
0000FFFF44FF03D0 0000FFFF80E38DA4 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
0000FFFF44FF0410 0000FFFF80909D58 System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(System.Threading.Tasks.Task)
0000FFFF44FF0430 0000FFFF80909C38 System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[[System.__Canon, System.Private.CoreLib]].GetResult()
0000FFFF44FF0460 0000FFFF80F87694 Elasticsearch.Net.RequestPipeline+<CallElasticsearchAsync>d__74`1[[System.__Canon, System.Private.CoreLib]].MoveNext() [C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs @ 467]
(lldb) clru 0000FFFF80E39A84
Normal JIT generated code
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
ilAddr is 0000FFFF7D47EF80 pImport is 000000003413EEF0
Begin 0000FFFF80E39A28, size 60
0000ffff80e39a28 fd7bbea9 stp x29, x30, [sp, #-0x20]!
0000ffff80e39a2c fd030091 mov x29, sp
0000ffff80e39a30 a00f00f9 str x0, [x29, #0x18]
0000ffff80e39a34 00c688d2 mov x0, #0x4630
0000ffff80e39a38 a0a3aff2 movk x0, #0x7d1d, lsl #16
0000ffff80e39a3c e0ffdff2 movk x0, #0xffff, lsl #32
0000ffff80e39a40 000040b9 ldr w0, [x0]
0000ffff80e39a44 40000034 cbz w0, 0xffff80e39a4c
0000ffff80e39a48 3e905897 bl 0xffff7e45db40
0000ffff80e39a4c 1f2003d5 nop
0000ffff80e39a50 a10f40f9 ldr x1, [x29, #0x18]
0000ffff80e39a54 3f0040b9 ldr wzr, [x1]
0000ffff80e39a58 a10f40f9 ldr x1, [x29, #0x18]
0000ffff80e39a5c 000280d2 mov x0, #0x10
0000ffff80e39a60 2100008b add x1, x1, x0
0000ffff80e39a64 a00f40f9 ldr x0, [x29, #0x18]
0000ffff80e39a68 000440f9 ldr x0, [x0, #0x8]
0000ffff80e39a6c 1f0040b9 ldr wzr, [x0]
0000ffff80e39a70 869b5897 bl 0xffff7e460888
0000ffff80e39a74 1f2003d5 nop
0000ffff80e39a78 a00f40f9 ldr x0, [x29, #0x18]
0000ffff80e39a7c 000440f9 ldr x0, [x0, #0x8]
0000ffff80e39a80 64905897 bl 0xffff7e45dc10
>>> 0000ffff80e39a84 000020d4 brk #0
Given that the IP for the ExceptionDispatchInfo.Throw() frame is at 0000ffff80e39a84, I assume we haven't returned from the call bl 0xffff7e45dc10
.
(lldb) clru 0xffff7e45dc10
Unmanaged code
0000ffff7e45dc10 50000058 ldr x16, #0x8
0000ffff7e45dc14 00021fd6 br x16
0000ffff7e45dc18 78031cf7 .long 0xf71c0378
0000ffff7e45dc1c ffff0000 udf #0xffff
Here it's a simple jump to 0xfffff71c0378. Which seems to be IL_Throw
:
(lldb) disassemble -s 0xfffff71c0378
libcoreclr.so`IL_Throw:
0xfffff71c0378 <+0>: str x28, [sp, #-0x20]!
0xfffff71c037c <+4>: stp x29, x30, [sp, #0x10]
0xfffff71c0380 <+8>: add x29, sp, #0x10 ; =0x10
0xfffff71c0384 <+12>: sub sp, sp, #0x2b0 ; =0x2b0
0xfffff71c0388 <+16>: mov x8, #0x0
0xfffff71c038c <+20>: adrp x9, 2134
0xfffff71c0390 <+24>: ldr x9, [x9, #0x878]
0xfffff71c0394 <+28>: ldr x9, [x9]
(lldb) disassemble -s 0xfffff71c0378 -c 300
libcoreclr.so`IL_Throw:
0xfffff71c0378 <+0>: str x28, [sp, #-0x20]!
0xfffff71c037c <+4>: stp x29, x30, [sp, #0x10]
0xfffff71c0380 <+8>: add x29, sp, #0x10 ; =0x10
0xfffff71c0384 <+12>: sub sp, sp, #0x2b0 ; =0x2b0
0xfffff71c0388 <+16>: mov x8, #0x0
0xfffff71c038c <+20>: adrp x9, 2134
0xfffff71c0390 <+24>: ldr x9, [x9, #0x878]
0xfffff71c0394 <+28>: ldr x9, [x9]
0xfffff71c0398 <+32>: stur x9, [x29, #-0x18]
0xfffff71c039c <+36>: str x0, [sp, #0xe8]
0xfffff71c03a0 <+40>: str x8, [sp, #0xe0]
0xfffff71c03a4 <+44>: bl 0xfffff72d0e04 ; ::ResetCurrentContext() at stubs.cpp:1240:1
0xfffff71c03a8 <+48>: str wzr, [sp, #0xdc]
0xfffff71c03ac <+52>: ldr x1, [sp, #0xe0]
0xfffff71c03b0 <+56>: add x8, sp, #0x190 ; =0x190
0xfffff71c03b4 <+60>: mov x0, x8
0xfffff71c03b8 <+64>: mov w2, #0x1
0xfffff71c03bc <+68>: str x8, [sp, #0x88]
0xfffff71c03c0 <+72>: bl 0xfffff6f2aac4 ; FrameWithCookie<HelperMethodFrame>::FrameWithCookie at frames.h:3236
0xfffff71c03c4 <+76>: ldr x0, [sp, #0x88]
0xfffff71c03c8 <+80>: bl 0xfffff6f2ab00 ; FrameWithCookie<HelperMethodFrame>::MachineState at frames.h:3292
0xfffff71c03cc <+84>: str x0, [sp, #0x80]
0xfffff71c03d0 <+88>: b 0xfffff71c03d4 ; <+92> at jithelpers.cpp
0xfffff71c03d4 <+92>: ldr x0, [sp, #0x80]
0xfffff71c03d8 <+96>: bl 0xfffff73de790 ; LazyMachStateCaptureState
0xfffff71c03dc <+100>: b 0xfffff71c03e0 ; <+104> at jithelpers.cpp
0xfffff71c03e0 <+104>: add x0, sp, #0x190 ; =0x190
0xfffff71c03e4 <+108>: adrp x1, 2188
0xfffff71c03e8 <+112>: add x1, x1, #0xefc ; =0xefc
0xfffff71c03ec <+116>: bl 0xfffff6f2ab4c ; FrameWithCookie<HelperMethodFrame>::SetAddrOfHaveCheckedRestoreState at frames.h:3274
0xfffff71c03f0 <+120>: b 0xfffff71c03f4 ; <+124> at jithelpers.cpp
0xfffff71c03f4 <+124>: add x0, sp, #0x170 ; =0x170
0xfffff71c03f8 <+128>: bl 0xfffff6ec4140 ; PAL_SEHException::PAL_SEHException at pal.h:4477
0xfffff71c03fc <+132>: b 0xfffff71c0400 ; <+136> at jithelpers.cpp
0xfffff71c0400 <+136>: mov w8, #0x0
0xfffff71c0404 <+140>: strb w8, [sp, #0xcb]
0xfffff71c0408 <+144>: add x0, sp, #0x190 ; =0x190
0xfffff71c040c <+148>: bl 0xfffff6f2ab80 ; FrameWithCookie<HelperMethodFrame>::Push at frames.h:3285
0xfffff71c0410 <+152>: b 0xfffff71c0414 ; <+156> at jithelpers.cpp
0xfffff71c0414 <+156>: add x0, sp, #0x190 ; =0x190
0xfffff71c0418 <+160>: bl 0xfffff6f2abac ; FrameWithCookie<HelperMethodFrame>::GetThread at frames.h:3293
0xfffff71c041c <+164>: str x0, [sp, #0x78]
0xfffff71c0420 <+168>: b 0xfffff71c0424 ; <+172> at jithelpers.cpp
0xfffff71c0424 <+172>: ldr x8, [sp, #0x78]
0xfffff71c0428 <+176>: str x8, [sp, #0xc0]
0xfffff71c042c <+180>: mov w9, #0x1
0xfffff71c0430 <+184>: strb w9, [sp, #0xbf]
0xfffff71c0434 <+188>: mov x10, #0x0
0xfffff71c0438 <+192>: str x10, [sp, #0xb0]
0xfffff71c043c <+196>: add x0, sp, #0x190 ; =0x190
0xfffff71c0440 <+200>: bl 0xfffff6f2abf8 ; FrameWithCookie<HelperMethodFrame>::operator& at frames.h:3291
0xfffff71c0444 <+204>: str x0, [sp, #0x70]
0xfffff71c0448 <+208>: b 0xfffff71c044c ; <+212> at jithelpers.cpp
0xfffff71c044c <+212>: ldr x8, [sp, #0x70]
0xfffff71c0450 <+216>: str x8, [sp, #0xa8]
0xfffff71c0454 <+220>: mov w9, #0x0
0xfffff71c0458 <+224>: strb w9, [sp, #0xa7]
0xfffff71c045c <+228>: ldr x1, [sp, #0xe8]
0xfffff71c0460 <+232>: add x0, sp, #0x140 ; =0x140
0xfffff71c0464 <+236>: bl 0xfffff702e430 ; OBJECTREF::OBJECTREF at object.cpp:1093
0xfffff71c0468 <+240>: b 0xfffff71c046c ; <+244> at jithelpers.cpp
0xfffff71c046c <+244>: add x0, sp, #0x138 ; =0x138
0xfffff71c0470 <+248>: mov x1, #0x0
0xfffff71c0474 <+252>: bl 0xfffff702e3c4 ; OBJECTREF::OBJECTREF at object.cpp:1068
0xfffff71c0478 <+256>: b 0xfffff71c047c ; <+260> at jithelpers.cpp
0xfffff71c047c <+260>: add x0, sp, #0x140 ; =0x140
0xfffff71c0480 <+264>: add x1, sp, #0x138 ; =0x138
0xfffff71c0484 <+268>: bl 0xfffff702e588 ; OBJECTREF::operator== at object.cpp:1140
0xfffff71c0488 <+272>: str w0, [sp, #0x6c]
0xfffff71c048c <+276>: b 0xfffff71c0490 ; <+280> at jithelpers.cpp
0xfffff71c0490 <+280>: ldr w8, [sp, #0x6c]
0xfffff71c0494 <+284>: cbz w8, 0xfffff71c04c8 ; <+336> at jithelpers.cpp
0xfffff71c0498 <+288>: mov w0, #0x2a
0xfffff71c049c <+292>: bl 0xfffff6f84ca8 ; RealCOMPlusThrow at excep.cpp:11584
0xfffff71c04a0 <+296>: b 0xfffff71c04a4 ; <+300> at jithelpers.cpp:4197:1
0xfffff71c04a4 <+300>: str x0, [sp, #0xd0]
0xfffff71c04a8 <+304>: str w1, [sp, #0xcc]
0xfffff71c04ac <+308>: b 0xfffff71c0830 ; <+1208> at jithelpers.cpp
0xfffff71c04b0 <+312>: str x0, [sp, #0xd0]
0xfffff71c04b4 <+316>: str w1, [sp, #0xcc]
0xfffff71c04b8 <+320>: b 0xfffff71c06c4 ; <+844> at jithelpers.cpp:4195:5
0xfffff71c04bc <+324>: str x0, [sp, #0xd0]
0xfffff71c04c0 <+328>: str w1, [sp, #0xcc]
0xfffff71c04c4 <+332>: b 0xfffff71c0540 ; <+456> at jithelpers.cpp:4195:5
0xfffff71c04c8 <+336>: add x0, sp, #0x140 ; =0x140
0xfffff71c04cc <+340>: bl 0xfffff702c264 ; OBJECTREF::operator-> at object.cpp:1218
0xfffff71c04d0 <+344>: str x0, [sp, #0x60]
0xfffff71c04d4 <+348>: b 0xfffff71c04d8 ; <+352> at jithelpers.cpp
0xfffff71c04d8 <+352>: ldr x0, [sp, #0x60]
0xfffff71c04dc <+356>: bl 0xfffff6eaeb50 ; Object::GetMethodTable at object.h:168
0xfffff71c04e0 <+360>: str x0, [sp, #0x58]
0xfffff71c04e4 <+364>: b 0xfffff71c04e8 ; <+368> at jithelpers.cpp
0xfffff71c04e8 <+368>: ldr x0, [sp, #0x58]
0xfffff71c04ec <+372>: bl 0xfffff6f73564 ; IsException at excep.cpp:8530
0xfffff71c04f0 <+376>: str w0, [sp, #0x54]
0xfffff71c04f4 <+380>: b 0xfffff71c04f8 ; <+384> at jithelpers.cpp
0xfffff71c04f8 <+384>: ldr w8, [sp, #0x54]
0xfffff71c04fc <+388>: cbnz w8, 0xfffff71c05ec ; <+628> at jithelpers.cpp:4183:13
0xfffff71c0500 <+392>: add x0, sp, #0x148 ; =0x148
0xfffff71c0504 <+396>: add x1, sp, #0x140 ; =0x140
0xfffff71c0508 <+400>: mov w2, #0x1
0xfffff71c050c <+404>: mov w3, #0x0
0xfffff71c0510 <+408>: bl 0xfffff6e87e6c ; GCFrame::GCFrame at frames.h:2405
0xfffff71c0514 <+412>: b 0xfffff71c0518 ; <+416> at jithelpers.cpp
0xfffff71c0518 <+416>: add x0, sp, #0x140 ; =0x140
0xfffff71c051c <+420>: bl 0xfffff6f73b94 ; WrapNonCompliantException at excep.cpp:443
0xfffff71c0520 <+424>: b 0xfffff71c0524 ; <+428> at jithelpers.cpp
0xfffff71c0524 <+428>: add x0, sp, #0x148 ; =0x148
0xfffff71c0528 <+432>: bl 0xfffff6f975dc ; GCFrame::~GCFrame at frames.cpp:962
0xfffff71c052c <+436>: b 0xfffff71c0690 ; <+792> at jithelpers.cpp
0xfffff71c0530 <+440>: str x0, [sp, #0xd0]
0xfffff71c0534 <+444>: str w1, [sp, #0xcc]
0xfffff71c0538 <+448>: add x0, sp, #0x148 ; =0x148
0xfffff71c053c <+452>: bl 0xfffff6f975dc ; GCFrame::~GCFrame at frames.cpp:962
0xfffff71c0540 <+456>: ldr w8, [sp, #0xcc]
0xfffff71c0544 <+460>: cmp w8, #0x2 ; =0x2
0xfffff71c0548 <+464>: b.ne 0xfffff71c06c4 ; <+844> at jithelpers.cpp:4195:5
0xfffff71c054c <+468>: ldr x0, [sp, #0xd0]
0xfffff71c0550 <+472>: bl 0xfffff6e36a70 ; symbol stub for: getpagesize
0xfffff71c0554 <+476>: str x0, [sp, #0x98]
0xfffff71c0558 <+480>: add x0, sp, #0xf0 ; =0xf0
0xfffff71c055c <+484>: bl 0xfffff6e6e100 ; CHECK::CHECK at check.h:117
0xfffff71c0560 <+488>: b 0xfffff71c0564 ; <+492> at jithelpers.cpp
0xfffff71c0564 <+492>: add x0, sp, #0xf0 ; =0xf0
0xfffff71c0568 <+496>: bl 0xfffff6e6e12c ; CHECK::EnterAssert at check.inl:13
0xfffff71c056c <+500>: str w0, [sp, #0x50]
0xfffff71c0570 <+504>: b 0xfffff71c0574 ; <+508> at jithelpers.cpp
0xfffff71c0574 <+508>: ldr w8, [sp, #0x50]
0xfffff71c0578 <+512>: cbz w8, 0xfffff71c073c ; <+964> at jithelpers.cpp:4195:5
0xfffff71c057c <+516>: ldr x8, [sp, #0x98]
0xfffff71c0580 <+520>: mov x9, #0x0
0xfffff71c0584 <+524>: cmp x9, x8
0xfffff71c0588 <+528>: cset w10, ne
0xfffff71c058c <+532>: mov w11, #0x1
0xfffff71c0590 <+536>: and w1, w10, w11
0xfffff71c0594 <+540>: add x0, sp, #0xf0 ; =0xf0
0xfffff71c0598 <+544>: bl 0xfffff6e6e46c ; CHECK::Fail at check.inl:79
0xfffff71c059c <+548>: str w0, [sp, #0x4c]
0xfffff71c05a0 <+552>: b 0xfffff71c05a4 ; <+556> at jithelpers.cpp
0xfffff71c05a4 <+556>: ldr w8, [sp, #0x4c]
0xfffff71c05a8 <+560>: cbz w8, 0xfffff71c0730 ; <+952> at jithelpers.cpp
0xfffff71c05ac <+564>: add x0, sp, #0xf0 ; =0xf0
0xfffff71c05b0 <+568>: adrp x1, 1443
0xfffff71c05b4 <+572>: add x1, x1, #0x1ce ; =0x1ce
0xfffff71c05b8 <+576>: adrp x2, 1411
0xfffff71c05bc <+580>: add x2, x2, #0x581 ; =0x581
0xfffff71c05c0 <+584>: adrp x3, 1465
0xfffff71c05c4 <+588>: add x3, x3, #0x99a ; =0x99a
0xfffff71c05c8 <+592>: mov w4, #0x1063
0xfffff71c05cc <+596>: bl 0xfffff73fecb8 ; CHECK::Setup at check.cpp:151
0xfffff71c05d0 <+600>: b 0xfffff71c05d4 ; <+604> at jithelpers.cpp
0xfffff71c05d4 <+604>: add x0, sp, #0xf0 ; =0xf0
0xfffff71c05d8 <+608>: adrp x1, 1405
0xfffff71c05dc <+612>: add x1, x1, #0x7df ; =0x7df
0xfffff71c05e0 <+616>: bl 0xfffff73fe808 ; CHECK::Trigger at check.cpp:105
0xfffff71c05e4 <+620>: b 0xfffff71c05e8 ; <+624> at jithelpers.cpp:4195:5
0xfffff71c05e8 <+624>: b 0xfffff71c0730 ; <+952> at jithelpers.cpp
0xfffff71c05ec <+628>: bl 0xfffff6e82c9c ; ::GetThread() at threads.inl:36
0xfffff71c05f0 <+632>: str x0, [sp, #0x40]
0xfffff71c05f4 <+636>: b 0xfffff71c05f8 ; <+640> at jithelpers.cpp
0xfffff71c05f8 <+640>: ldr x0, [sp, #0x40]
0xfffff71c05fc <+644>: bl 0xfffff6f87924 ; Thread::GetExceptionState at threads.h:2259
0xfffff71c0600 <+648>: str x0, [sp, #0x38]
0xfffff71c0604 <+652>: b 0xfffff71c0608 ; <+656> at jithelpers.cpp
0xfffff71c0608 <+656>: ldr x0, [sp, #0x38]
0xfffff71c060c <+660>: bl 0xfffff6f8798c ; ThreadExceptionState::IsRaisingForeignException at exstate.h:127
0xfffff71c0610 <+664>: str w0, [sp, #0x34]
0xfffff71c0614 <+668>: b 0xfffff71c0618 ; <+672> at jithelpers.cpp
0xfffff71c0618 <+672>: ldr w8, [sp, #0x34]
0xfffff71c061c <+676>: cbz w8, 0xfffff71c0664 ; <+748> at jithelpers.cpp
0xfffff71c0620 <+680>: add x0, sp, #0x130 ; =0x130
0xfffff71c0624 <+684>: add x1, sp, #0x140 ; =0x140
0xfffff71c0628 <+688>: bl 0xfffff6e83034 ; REF<ExceptionObject>::REF at vars.hpp:246
0xfffff71c062c <+692>: b 0xfffff71c0630 ; <+696> at jithelpers.cpp
0xfffff71c0630 <+696>: add x0, sp, #0x130 ; =0x130
0xfffff71c0634 <+700>: bl 0xfffff6e83060 ; REF<ExceptionObject>::operator-> at vars.hpp:271
0xfffff71c0638 <+704>: str x0, [sp, #0x28]
0xfffff71c063c <+708>: b 0xfffff71c0640 ; <+712> at jithelpers.cpp
0xfffff71c0640 <+712>: add x0, sp, #0x128 ; =0x128
0xfffff71c0644 <+716>: mov x1, #0x0
0xfffff71c0648 <+720>: bl 0xfffff6f3b0d8 ; REF<StringObject>::REF at vars.hpp:256
0xfffff71c064c <+724>: b 0xfffff71c0650 ; <+728> at jithelpers.cpp
0xfffff71c0650 <+728>: ldr x0, [sp, #0x28]
0xfffff71c0654 <+732>: add x1, sp, #0x128 ; =0x128
0xfffff71c0658 <+736>: bl 0xfffff6f87a24 ; ExceptionObject::SetStackTraceString at object.h:2491
0xfffff71c065c <+740>: b 0xfffff71c0660 ; <+744> at jithelpers.cpp:4186:9
0xfffff71c0660 <+744>: b 0xfffff71c0690 ; <+792> at jithelpers.cpp
0xfffff71c0664 <+748>: add x0, sp, #0x120 ; =0x120
0xfffff71c0668 <+752>: add x1, sp, #0x140 ; =0x140
0xfffff71c066c <+756>: bl 0xfffff6e83034 ; REF<ExceptionObject>::REF at vars.hpp:246
0xfffff71c0670 <+760>: b 0xfffff71c0674 ; <+764> at jithelpers.cpp
0xfffff71c0674 <+764>: add x0, sp, #0x120 ; =0x120
0xfffff71c0678 <+768>: bl 0xfffff6e83060 ; REF<ExceptionObject>::operator-> at vars.hpp:271
0xfffff71c067c <+772>: str x0, [sp, #0x20]
0xfffff71c0680 <+776>: b 0xfffff71c0684 ; <+780> at jithelpers.cpp
0xfffff71c0684 <+780>: ldr x0, [sp, #0x20]
0xfffff71c0688 <+784>: bl 0xfffff71cb474 ; ExceptionObject::ClearStackTracePreservingRemoteStackTrace at object.h:2529
0xfffff71c068c <+788>: b 0xfffff71c0690 ; <+792> at jithelpers.cpp
0xfffff71c0690 <+792>: add x0, sp, #0x118 ; =0x118
0xfffff71c0694 <+796>: add x1, sp, #0x140 ; =0x140
0xfffff71c0698 <+800>: bl 0xfffff702e2e0 ; OBJECTREF::OBJECTREF at object.cpp:1034
0xfffff71c069c <+804>: b 0xfffff71c06a0 ; <+808> at jithelpers.cpp
0xfffff71c06a0 <+808>: add x0, sp, #0x118 ; =0x118
0xfffff71c06a4 <+812>: mov w8, #0x0
0xfffff71c06a8 <+816>: mov w1, w8
0xfffff71c06ac <+820>: mov w2, w8
0xfffff71c06b0 <+824>: bl 0xfffff6f78f78 ; RaiseTheExceptionInternalOnly at excep.cpp:2670
0xfffff71c06b4 <+828>: b 0xfffff71c06b8 ; <+832> at jithelpers.cpp:4197:1
0xfffff71c06b8 <+832>: str x0, [sp, #0xd0]
0xfffff71c06bc <+836>: str w1, [sp, #0xcc]
0xfffff71c06c0 <+840>: bl 0xfffff6e37510 ; symbol stub for: strcmp
0xfffff71c06c4 <+844>: ldr w8, [sp, #0xcc]
0xfffff71c06c8 <+848>: cmp w8, #0x1 ; =0x1
0xfffff71c06cc <+852>: b.ne 0xfffff71c0828 ; <+1200> at jithelpers.cpp
0xfffff71c06d0 <+856>: ldr x0, [sp, #0xd0]
0xfffff71c06d4 <+860>: bl 0xfffff6e36a70 ; symbol stub for: getpagesize
0xfffff71c06d8 <+864>: str x0, [sp, #0x90]
0xfffff71c06dc <+868>: ldr x0, [sp, #0x90]
0xfffff71c06e0 <+872>: bl 0xfffff6ec423c ; std::move<PAL_SEHException&> at utility:30
0xfffff71c06e4 <+876>: str x0, [sp, #0x18]
0xfffff71c06e8 <+880>: b 0xfffff71c06ec ; <+884> at jithelpers.cpp
0xfffff71c06ec <+884>: add x0, sp, #0x170 ; =0x170
0xfffff71c06f0 <+888>: ldr x1, [sp, #0x18]
0xfffff71c06f4 <+892>: bl 0xfffff6ec4250 ; PAL_SEHException::operator= at pal.h:4493
0xfffff71c06f8 <+896>: b 0xfffff71c06fc ; <+900> at jithelpers.cpp
0xfffff71c06fc <+900>: mov w8, #0x1
0xfffff71c0700 <+904>: strb w8, [sp, #0xcb]
0xfffff71c0704 <+908>: bl 0xfffff6e37510 ; symbol stub for: strcmp
0xfffff71c0708 <+912>: b 0xfffff71c070c ; <+916> at jithelpers.cpp:4195:5
0xfffff71c070c <+916>: ldrb w8, [sp, #0xcb]
0xfffff71c0710 <+920>: tbnz w8, #0x0, 0xfffff71c0718 ; <+928> at jithelpers.cpp
0xfffff71c0714 <+924>: b 0xfffff71c07ac ; <+1076> at jithelpers.cpp
0xfffff71c0718 <+928>: add x0, sp, #0x170 ; =0x170
0xfffff71c071c <+932>: mov w8, #0x1
0xfffff71c0720 <+936>: mov w9, #0x0
0xfffff71c0724 <+940>: and w1, w9, w8
0xfffff71c0728 <+944>: bl 0xfffff72e19bc ; DispatchManagedException at exceptionhandling.cpp:4648
0xfffff71c072c <+948>: b 0xfffff71c0730 ; <+952> at jithelpers.cpp
0xfffff71c0730 <+952>: add x0, sp, #0xf0 ; =0xf0
0xfffff71c0734 <+956>: bl 0xfffff6e6e450 ; CHECK::LeaveAssert at check.inl:35
0xfffff71c0738 <+960>: b 0xfffff71c073c ; <+964> at jithelpers.cpp:4195:5
0xfffff71c073c <+964>: ldr x8, [sp, #0x98]
0xfffff71c0740 <+968>: str x8, [sp, #0xb0]
0xfffff71c0744 <+972>: ldr x0, [sp, #0xa8]
0xfffff71c0748 <+976>: ldr x1, [sp, #0xb0]
0xfffff71c074c <+980>: bl 0xfffff6f7e7d0 ; UnwindAndContinueRethrowHelperInsideCatch at excep.cpp:8083
0xfffff71c0750 <+984>: b 0xfffff71c0754 ; <+988> at jithelpers.cpp
0xfffff71c0754 <+988>: mov w8, #0x1
0xfffff71c0758 <+992>: strb w8, [sp, #0xa7]
0xfffff71c075c <+996>: bl 0xfffff6e37510 ; symbol stub for: strcmp
0xfffff71c0760 <+1000>: ldrb w8, [sp, #0xa7]
0xfffff71c0764 <+1004>: tbnz w8, #0x0, 0xfffff71c076c ; <+1012> at jithelpers.cpp:4195:5
0xfffff71c0768 <+1008>: b 0xfffff71c077c ; <+1028> at jithelpers.cpp
0xfffff71c076c <+1012>: ldr x0, [sp, #0xa8]
0xfffff71c0770 <+1016>: ldr x1, [sp, #0xb0]
0xfffff71c0774 <+1020>: bl 0xfffff6f7e96c ; UnwindAndContinueRethrowHelperAfterCatch at excep.cpp:8122
0xfffff71c0778 <+1024>: b 0xfffff71c077c ; <+1028> at jithelpers.cpp
0xfffff71c077c <+1028>: add x0, sp, #0x190 ; =0x190
0xfffff71c0780 <+1032>: bl 0xfffff6f2adb0 ; FrameWithCookie<HelperMethodFrame>::Pop at frames.h:3286
0xfffff71c0784 <+1036>: b 0xfffff71c0788 ; <+1040> at jithelpers.cpp:4195:5
0xfffff71c0788 <+1040>: b 0xfffff71c070c ; <+916> at jithelpers.cpp:4195:5
0xfffff71c078c <+1044>: str x0, [sp, #0xd0]
0xfffff71c0790 <+1048>: str w1, [sp, #0xcc]
0xfffff71c0794 <+1052>: bl 0xfffff6e37510 ; symbol stub for: strcmp
0xfffff71c0798 <+1056>: b 0xfffff71c07a8 ; <+1072> at jithelpers.cpp:4195:5
0xfffff71c079c <+1060>: str x0, [sp, #0xd0]
0xfffff71c07a0 <+1064>: str w1, [sp, #0xcc]
0xfffff71c07a4 <+1068>: b 0xfffff71c0828 ; <+1200> at jithelpers.cpp
0xfffff71c07a8 <+1072>: b 0xfffff71c0828 ; <+1200> at jithelpers.cpp
0xfffff71c07ac <+1076>: add x0, sp, #0x170 ; =0x170
0xfffff71c07b0 <+1080>: bl 0xfffff6ec4290 ; PAL_SEHException::~PAL_SEHException at pal.h:4500
0xfffff71c07b4 <+1084>: add x0, sp, #0x190 ; =0x190
0xfffff71c07b8 <+1088>: bl 0xfffff6f2abf8 ; FrameWithCookie<HelperMethodFrame>::operator& at frames.h:3291
0xfffff71c07bc <+1092>: str x0, [sp, #0x10]
0xfffff71c07c0 <+1096>: b 0xfffff71c07c4 ; <+1100> at jithelpers.cpp
0xfffff71c07c4 <+1100>: add x0, sp, #0x190 ; =0x190
0xfffff71c07c8 <+1104>: bl 0xfffff6f2ab00 ; FrameWithCookie<HelperMethodFrame>::MachineState at frames.h:3292
0xfffff71c07cc <+1108>: str x0, [sp, #0x8]
0xfffff71c07d0 <+1112>: b 0xfffff71c07d4 ; <+1116> at jithelpers.cpp
0xfffff71c07d4 <+1116>: ldr x0, [sp, #0x10]
0xfffff71c07d8 <+1120>: ldr x1, [sp, #0x8]
0xfffff71c07dc <+1124>: bl 0xfffff73de7c4 ; HelperMethodFrameRestoreState
0xfffff71c07e0 <+1128>: str w0, [sp, #0x4]
0xfffff71c07e4 <+1132>: b 0xfffff71c07e8 ; <+1136> at jithelpers.cpp
0xfffff71c07e8 <+1136>: ldr w8, [sp, #0x4]
0xfffff71c07ec <+1140>: str w8, [sp, #0xdc]
0xfffff71c07f0 <+1144>: add x0, sp, #0x190 ; =0x190
0xfffff71c07f4 <+1148>: bl 0xfffff6f2addc ; FrameWithCookie<HelperMethodFrame>::~FrameWithCookie at frames.h:3169
0xfffff71c07f8 <+1152>: ldr w8, [sp, #0xdc]
0xfffff71c07fc <+1156>: cbnz w8, 0xfffff71c03ac ; <+52> at jithelpers.cpp:4157:5
0xfffff71c0800 <+1160>: adrp x8, 2134
0xfffff71c0804 <+1164>: ldr x8, [x8, #0x878]
0xfffff71c0808 <+1168>: ldr x8, [x8]
0xfffff71c080c <+1172>: ldur x9, [x29, #-0x18]
0xfffff71c0810 <+1176>: cmp x8, x9
0xfffff71c0814 <+1180>: b.ne 0xfffff71c0844 ; <+1228> at jithelpers.cpp
0xfffff71c0818 <+1184>: add sp, sp, #0x2b0 ; =0x2b0
0xfffff71c081c <+1188>: ldp x29, x30, [sp, #0x10]
0xfffff71c0820 <+1192>: ldr x28, [sp], #0x20
0xfffff71c0824 <+1196>: ret
Unfortunately there are too many branches in this method, so I got lost pretty quickly. But that made me think: how can we make it all the way to a virtual dispatch stub if we still haven't returned from IL_Throw
? That's a wild guess, but could it be an exception filter?
We're starting to get far enough in the details that I don't have an immediate answer.
@jkotas is there anyone who is familiar with exception handling that could take a look? If not I can dig further but it will take me a bit to dig in and get an answer.
The original problem tracked by this issue (SegFault in AdjustContextForVirtualStub) is fixed. We should open a new issue to discuss the other AV.
Everything in the description suggests that interface method was called on NULL pointer, and the exception was caught and handled gracefully. The only outlier is the output from clrstack
. My guess is that clrstack
is lying for some reason.
I have an application that consistently crashes only on Linux/ARM64, with the following stacktrace:
This is an application that throws a bunch of exceptions, on top of lengthy callstacks (this one is 153 frames long).
It's important to note that the crash only occurs when our profiler is attached and we've ReJITted some methods. So it's entirely possible we're corrupting something. Still, the consistent location of the error and the fact that it happens only on ARM64 is fishy.
Using .NET 5.0.3.
I'll keep digging on my side to figure out if the issue is in the runtime or in our profiler. coredump.zip