adamrodger
commented
9 months ago Just to confirm our theory that the segfault happens during GC, I added the following at the end of the loop after the interop code has been disposed/freed:
GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced, blocking: true, compacting: true)
GC.Collect(GC.MaxGeneration, GCCollectionMode.Aggressive, blocking: true, compacting: true)The segfault is still intermittent, but now it happens during the Aggressive GC (the second one). The core dump name has now changed also. Now the dump is named core..NET Finalizer.8 and the thread backtrace is
* thread #1, name = 'dotnet', stop reason = signal SIGSEGV
* frame #0: 0x00007fec3fa2f98a libcoreclr.so`IsIPInEpilog(_CONTEXT*, EECodeInfo*, int*) [inlined] IsIPInProlog(pCodeInfo=0x00007fec3cb51328) at excep.cpp:7169:49
frame #1: 0x00007fec3fa2f987 libcoreclr.so`IsIPInEpilog(pContextToCheck=0x00007fec3cb52e60, pCodeInfo=0x00007fec3cb51328, pSafeToInjectThreadAbort=YES) at excep.cpp:7236:9
frame #2: 0x00007fec3fbd9ca4 libcoreclr.so`HandleSuspensionForInterruptedThread(interruptedContext=0x00007fec3cb52e60) at threadsuspend.cpp:5914:13
frame #3: 0x00007fec3fe0f08c libcoreclr.so`inject_activation_handler(code=<unavailable>, siginfo=<unavailable>, context=0x00007fec3cb53ac0) at signal.cpp:840:13
frame #4: 0x00007fec3fff9050 libc.so.6`___lldb_unnamed_symbol3252 + 1
frame #5: 0x00007febc0bd7aea
frame #6: 0x00007fec3fcc7ba6 libcoreclr.so`FastCallFinalizeWorker at calldescrworkeramd64.S:30
frame #7: 0x00007fec3fa84213 libcoreclr.so`MethodTable::CallFinalizer(Object*) at methodtable.cpp:4770:5
frame #8: 0x00007fec3fa841c9 libcoreclr.so`MethodTable::CallFinalizer(obj=0x00007feb92834630) at methodtable.cpp:4888:5
frame #9: 0x00007fec3fb3d955 libcoreclr.so`FinalizerThread::FinalizeAllObjects() [inlined] CallFinalizer(obj=0x00007feb92834630) at finalizerthread.cpp:75:9
frame #10: 0x00007fec3fb3d8fd libcoreclr.so`FinalizerThread::FinalizeAllObjects() at finalizerthread.cpp:104:9
frame #11: 0x00007fec3fb3dba5 libcoreclr.so`FinalizerThread::FinalizerThreadWorker(args=<unavailable>) at finalizerthread.cpp:348:9
frame #12: 0x00007fec3face7c5 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [inlined] ManagedThreadBase_DispatchInner(pCallState=<unavailable>) at threads.cpp:7222:5
frame #13: 0x00007fec3face7c3 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at threads.cpp:7266:9
frame #14: 0x00007fec3face788 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [inlined] ManagedThreadBase_DispatchOuter(this=<unavailable>, pParam=<unavailable>)::$_0::operator()(ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::TryArgs*) const::'lambda'(Param*)::operator()(Param*) const at threads.cpp:7424:13
frame #15: 0x00007fec3face788 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at threads.cpp:7426:9
frame #16: 0x00007fec3face765 libcoreclr.so`ManagedThreadBase_DispatchOuter(pCallState=0x00007fec3d353de0) at threads.cpp:7450:5
frame #17: 0x00007fec3facedcd libcoreclr.so`ManagedThreadBase::FinalizerBase(void (*)(void*)) [inlined] ManagedThreadBase_NoADTransition(pTarget=<unavailable>, filterType=FinalizerThread)(void*), UnhandledExceptionLocation) at threads.cpp:7494:5
frame #18: 0x00007fec3facedb5 libcoreclr.so`ManagedThreadBase::FinalizerBase(pTarget=<unavailable>)(void*)) at threads.cpp:7513:5
frame #19: 0x00007fec3fb3de48 libcoreclr.so`FinalizerThread::FinalizerThreadStart(args=<unavailable>) at finalizerthread.cpp:398:17
frame #20: 0x00007fec3fe3effe libcoreclr.so`CorUnix::CPalThread::ThreadEntry(pvParam=0x000055accab9f510) at thread.cpp:1760:16
frame #21: 0x00007fec40046134 libc.so.6`___lldb_unnamed_symbol3514 + 708
frame #22: 0x00007fec400c67dc libc.so.6`___lldb_unnamed_symbol3939 + 11
jkotas
rmsimpson
alexrp
huoyaoyuan
me-viper
teh13th
agocke
AaronRobinsonMSFT
dotnet-policy-service[bot]
RoadTrain
MichalPetryka
Description
I have a C# application which does interop with a native C++ library which has started to intermittently crash with a segmentation fault since upgrading to .Net 8. The code is unmodified from prior to the upgrade, and has worked in the past on .Net 5, 6 and 7 in production use without a segfault.
Reproduction Steps
I have written a few smaller apps to reproduce the issue, but it's difficult to share them given they rely on calling the proprietary C++ library.
Essentially the apps:
The segfault always happens during the data load step (i.e. when a large amount of managed memory is being allocated), but only once the interop has happened at least once. I've verified stubbing out the interop calls and then the app never segfaults.
Expected behavior
The app works as per .Net 7 version.
Actual behavior
The app crashes with a segfault intermittently.
Regression?
Yes, the app worked on .Net 7 and I can change the version back to .Net 7 and it continues to work fine.
Known Workarounds
No known workarounds other than not using .Net 8, although the problem does get worse the less memory the container has. Our theory is that this triggers more frequent garbage collections and this is where the segfault occurs.
Configuration
.Net Version: 8.0.2 OS: Linux (Debian bookworm) Container image:
mcr.microsoft.com/dotnet/sdk:8.0.2(used for analysing core dumps) and also inmcr.microsoft.com/dotnet/runtime:8.0.2in production Architecture: x86-64Other information
LLDB backtrace:
LLDB frame variable:
I've tried to configure
createdumpto capture core dumps when the application crashes but it never seems to trigger (it does on other apps). I assume the segfault prevents thecreatedumpfrom triggering so I only have the standard core dump written by Linux. I've tried to get the CLR stack output but the core dump doesn't seem to work with SOS.I've also run the test apps on Windows and I can never get them to segfault. They only seem to segfault on Linux.