search

microsoft / clrmd

Microsoft.Diagnostics.Runtime is a set of APIs for introspecting processes and dumps.
MIT License
1.05k stars 255 forks source link

System.AccessViolationException when reading dump #947

Open PhDuck opened 3 years ago

PhDuck commented 3 years ago 
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Repeat 2 times:
--------------------------------
   at Microsoft.Diagnostics.Runtime.Windows.CacheNativeMethods+Memory.memcpy(IntPtr, UIntPtr, UIntPtr)
--------------------------------
   at Microsoft.Diagnostics.Runtime.Windows.ArrayPoolBasedCacheEntry.CopyDataFromPage(Microsoft.Diagnostics.Runtime.Windows.CachePage`1<Byte[]>, IntPtr, UInt32, UInt32)
   at Microsoft.Diagnostics.Runtime.Windows.CacheEntryBase`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ReadPageDataFromOffset(Int32, UInt32, UInt32, IntPtr, System.Func`3<UIntPtr,UInt32
,UInt32>)
   at Microsoft.Diagnostics.Runtime.Windows.CacheEntryBase`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetDataForAddress(UInt64, UInt32, IntPtr, UInt32 ByRef)
   at Microsoft.Diagnostics.Runtime.Windows.CachedMemoryReader.TryReadMemory(UInt64, Int32, IntPtr)
   at Microsoft.Diagnostics.Runtime.Windows.CachedMemoryReader.Read(UInt64, System.Span`1<Byte>)
   at Microsoft.Diagnostics.Runtime.MemoryReader.MoveToPage(UInt64)
   at Microsoft.Diagnostics.Runtime.MemoryReader.ReadPtr(UInt64, UInt64 ByRef)
   at Microsoft.Diagnostics.Runtime.Implementation.ClrmdSegment+<EnumerateObjects>d__39.MoveNext()
   at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.Diagnostics.Runtime.ClrObject, Microsoft.Diagnostics.Runtime, Version
=2.0.4.26801, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].MoveNext()
   at System.Linq.Enumerable+WhereEnumerableIterator`1[[Microsoft.Diagnostics.Runtime.ClrObject, Microsoft.Diagnostics.Runtime, Version=2.0.4.26801, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].MoveNext()
   at MemAna.Analysis.Rules.StringDuplication.ObjectWalk()
   at MemAna.Analysis.Rules.StringDuplication.ExecuteImpl()
   at MemAna.Analysis.AnalysisRule+<>c__DisplayClass6_0.<Execute>b__0()
   at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
   at System.Threading.Tasks.Task+<>c.<.cctor>b__277_0(System.Object)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
   at System.Threading.Tasks.Task.ExecuteFromThreadPool(System.Threading.Thread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

OS: Windows Server 2016 (Azure VM if that matters) 14393.4402.amd64fre.rs1_release.210426-1725 CoreCLR Version: 5.0.921.35908 .NET Version: 5.0.9 X64 Microsoft.Diagnostics.Runtime: 2.0.226801

Dump info: .NET 4.8 X64 32GB

Seems to crash when hitting around 2.3 GB of memory usage, which seems a bit strange.

PhDuck commented 3 years ago

If I set new CacheOptions() { MaxDumpCacheSize = 1} when loading the dump it no longer crashes but is just very slow... as warned: Minidump.cs::112 // this will be very slow

I guess the CachedMemoryReader is not cleaning up after itself?

PhDuck commented 3 years ago

If it helps here is the !DumpHeap -stat view from the crash dump of the exe using ClrMD:

00007ffd91be7e90       37        22374 System.Char[]
00007ffd91bed788      221        22984 Microsoft.Diagnostics.Runtime.ModuleInfo
00007ffd91a6ed20        1        23892 Microsoft.Diagnostics.Runtime.Windows.MinidumpModule[]
00007ffd91e5f2d8      987        39480 Microsoft.Win32.SafeHandles.SafeMemoryMappedViewHandle
00007ffd91e5f8c0      987        47376 System.IO.MemoryMappedFiles.MemoryMappedView
00007ffd91ecf840        2        52816 System.ValueTuple`2[[System.UInt64, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]][]
00007ffd91e5ef28      987        55272 System.IO.MemoryMappedFiles.MemoryMappedViewAccessor
00007ffd919824a8     1112        60984 System.SByte[]
00007ffd918c6860       49        61512 System.Object[]
00007ffd91e85b90      530        80560 Microsoft.Diagnostics.Runtime.Implementation.ClrmdArrayType
00007ffd91e7b5e8       10       201512 Microsoft.Diagnostics.Runtime.ClrModule[]
00007ffd91e864f0        2       299280 System.Collections.Generic.Dictionary`2+Entry[[System.UInt64, System.Private.CoreLib],[Microsoft.Diagnostics.Runtime.ClrType, Microsoft.Diagnostics.Runtime]][]
00007ffd91e85820     3855       524280 Microsoft.Diagnostics.Runtime.Implementation.ClrmdType
00007ffd91e7cf18        1       872496 System.Collections.Generic.Dictionary`2+Entry[[System.UInt64, System.Private.CoreLib],[Microsoft.Diagnostics.Runtime.ClrModule, Microsoft.Diagnostics.Runtime]][]
00007ffd91a7f878        3      1610616 Microsoft.Diagnostics.Runtime.Windows.MinidumpSegment[]
00007ffd91ad50d0        1      1810368 System.Collections.Generic.Dictionary`2+Entry[[System.UInt64, System.Private.CoreLib],[Microsoft.Diagnostics.Runtime.Windows.SegmentCacheEntry, Microsoft.Diagnostics.Runtime]][]
00007ffd91982990       36      2435228 System.Int32[]
00007ffd91e7c0d0    25143      3821736 Microsoft.Diagnostics.Runtime.Implementation.ClrmdModule
00007ffd91ad3b48    67106      4294784 System.Action`1[[System.UInt32, System.Private.CoreLib]]
00007ffd91ad43b0    67106      5905328 Microsoft.Diagnostics.Runtime.Windows.ArrayPoolBasedCacheEntry
00007ffd91ad4cd8   231193      7398176 Microsoft.Diagnostics.Runtime.Windows.CachePage`1[[System.Byte[], System.Private.CoreLib]]
00007ffd91ece1d8        2     15388400 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.ValueTuple`2[[System.UInt64, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]], System.Private.CoreLib]][]
00007ffd91987a90   289208     48809726 System.String
000001cc8025eb70   436572     63287800      Free
00007ffd91ad4d78    67106     67116720 Microsoft.Diagnostics.Runtime.Windows.CachePage`1[[System.Byte[], System.Private.CoreLib]][]
00007ffd91ad4a38    67106     67116720 System.Threading.ReaderWriterLockSlim[]
00007ffd91ad1f18  8188273    786074208 System.Threading.ReaderWriterLockSlim
00007ffd919cbf30   231536    952648258 System.Byte[]
ryanmolden commented 1 year ago

@PhDuck thanks for the report (I know its quite old, I never use GitHub so I am just seeing this :().

I wrote the code in question so I can certainly take a look, though it looks like @catstrike may have found the issue if the segment size is > int32.MaxValue.

One alternative you could try, though it is Windows only and requires enabling a specific setting, would be to use the AWE setting instead of the array style caching.

I think you enable this by setting the CacheOptions to UseOSMemoryFeatures (the ones passed in to the minidump ctor) and then setting up Windows to enable locking pages in memory (requires a reboot):

https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/enable-the-lock-pages-in-memory-option-windows?view=sql-server-ver16