Open mgasparel opened 3 years ago
Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.
Tagging subscribers to this area: @dotnet/gc See info in area-owners.md if you want to be subscribed.
@mgasparel I don't have a CLR version- release version mapping handy. What version of .NET Core / .NET 5 is this?
@danmosemsft This should be 3.1.7
@mgasparel, where is the dump?
@mgasparel one possible secure way to submit a dump to @cshung is to open a dummy feedback issue here https://developercommunity.visualstudio.com/spaces/61/index.html and share the ID.
@danmosemsft Our security team has indicated that we can not share the raw dump. Is there anything we can do without sharing the dump? Any specific commands/analysis you would want to run? I'm happy to be your remote hands and provide output/results.
@mgasparel i will let @cshung guide you as he’s the area owner.
I had a call with Mike, and he indicated that:
For now, there is not much we can do, and we are closing this issue for now. We can revisit it if it happens again or when we have more information to work on.
The customer reported back that he can access the dump again. We will take a look and see if it is actionable.
Just wanted to let you know that we had this issue on another server this week
PerfView arguments:
PerfView.exe collect -CollectMultiple=1000000 -MaxCollectSec=10 -AcceptEULA -NoView -NoGui -CircularMB=1024 -BufferSize=1024 -CPUSampleMSec:10 -ClrEvents=JITSymbols+GC+GCHeapSurvivalAndMovement+Stack+Exception -KernelEvents=process+thread+ImageLoad+Profile+ThreadTime -DotNetAllocSampled -NoNGenRundown -NoV2Rundown -LogFile:perfviewlog-$timestamp.txt -DataFile:PerfView-$timestamp.etl
@cshung
The bug is understood.
When a new ETW session is established, coreclr!EtwCallback
will be called on a thread. Under certain conditions, coreclr!EtwCallback
calls coreclr!ETW::GCLog::GCSettingsEvent
which then calls GCHeap::DiagTraceGCSegments
. It will access the segment list without synchronizing with the GC. By chance, we might hit a case where the GC alters the segment list (e.g. when the GC decommit a segment), and that would cause an access violation as we witness in this issue.
With regard to whether or not it will repro with perfcollect
, can you check if you have a similar call to coreclr!ETW::GCLog::GCSettingsEvent
when perfcollect
happens?
@dotnet/dotnet-diag
@cshung Could you please confirm if this is considered PerfView issue or the framework's (GC) or both? If the framework's: is it going to be resolved by 6.0, or earlier in the version 5 of the framework?
With regard to whether or not it will repro with perfcollect, can you check if you have a similar call to coreclr!ETW::GCLog::GCSettingsEvent when perfcollect happens?
As far as I can tell GCLog::GCSettingsEvent
is only called from EtwCallback
. Using perfcollect
or EventPipe does not trigger the runtime to call back into EtwCallback
as this callback is specifically for ETW only.
@baal2000 The bug is in the runtime. @mangod9 can help with the question about when and where we will deliver the fix.
@mangod9 Is there an ETA on this issue update?
Hi @baal2000, we hope to get to it in the next few .net 6 previews. Hope that works for you?
@mangod9 Ideally this is fixed in version 5 because preview frameworks are not for production use. The hope is that the fix is narrow enough to not represent much of a regression risk. Thank you.
Yeah once we have a fix ready we can determine risk of a backport. Thx.
@cshung, is this issue and/or fix well understood?
Moving to 7 since this is a corner case.
I don't think we have time for this 7.0. moving this out; let me know if you disagree.
(sorry accidentally closed)
I disagree this is a corner case.
Hi,
We rolled out PerfView to help with this issue and one of the things we've seen is that PerfView on occasion itself causes the app the crash:
This isn't great. Probably happens once every two weeks.
I don't know if it is the runtime or PerfView at fault here.
Cheers, Indy
Description
This week our application crashed with an access violation error. We were able to retrieve an automatic crash dump, and it appears that this exception originated in the GC at
coreclr!SVR::GCHeap::DiagTraceGCSegments+0x94 [f:\workspace\_work\1\s\src\gc\gcee.cpp @ 444]
Configuration
CLR Version: 4.700.20.36602
OS_VERSION: 10.0.14393.2430
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
Other information
I did some digging around in this repos issues and could not find anything that seemed relevant to this case.
crash dump analysis from windbg below:
Happy to assist if you need any further information or analysis on the dump.