cshung
commented
5 years ago Thanks @janvorli for the investigation, he pointed out that problem is the code is doing something inappropriate according to the contracts. In particular, UnlockedLoaderHeap::UnlockedAllocMem indicated it is a GC_NOTRIGGER region yet later on we did a GCX_MAYBE_COOP_NO_THREAD_BROKEN later on and caused grief.
Later on we discussed around and identified 4 possible solutions:
- Ensure WriteEvent() is only called in places where StackWalk is safe.
- If we could detect StackWalk is not safe, skip the stack, or
- Make StackWalk safe.
- Replace managed stack walk with a native stack walker.
Option 1 is very tedious, if not impossible. EtmDummy.h indicates we have about 400 different call sites for WriteEvent.
Option 2 is also tedious, there were attempts trying to detect whether or not StackWalk is safe. It is hard to do in general, and is also error prone.
Option 3 is interesting - so far the only offender I am seeing (in my simple stack) is ReadyToRunInfo::GetMethodDescForEntryPoint trying to do a hash table lookup, and for reason that I don't understand, it enters into cooperative mode, and screw us up. It could be the only case, or it could a tip of the iceberg, I have no idea for now. If we could make stack walking never need to enter cooperative mode, we can ensure a robust implementation of EventPipe. The great thing about this approach is that there is one big component to analyze, but not 400 call sites.
Option 4 is interesting too, if we use a different stack walker, then we are break free from the constraint of the current stack walker. It also allow the EventPipe traces to get native frames for analysis. It will likely not hit the time we need for .NET Core 3, and there might be other reasons that I don't know to introduce a native stack walker into the runtime.
@jkotas @fadimounir - for the ready to run use case @AaronRobinsonMSFT - for what might that RCW reason be. @sywhang @noahfalk @jorive @vancem @tommcdon
AaronRobinsonMSFT
jkotas
vancem
noahfalk
davidwrighton
tommcdon
Symptom:
When debugging a test case with an aggresive printing breakpoint set, the debuggee deadlocks.
Repro (x86, debug)
Run the test case in dotnet/runtime#12180 with WinDBG attached and set this aggressively logging breakpoint
bm CoreCLR!EventPipeBufferList::InsertTail "k;g". It will stop and deadlock.Analysis
It appears that the EE suspension triggered in thread 6 will not complete because thread 9 is waiting on the LoaderHeap lock while it is held by thread 0 which is already suspended.
@janvorli, can you please take a look?
Stacks