Closed myarichuk closed 4 years ago
Looking at dump dotnet/coreclr#3, the heap is corrupted:
0:077> !VerifyHeap
object 000001f1b1bc0108: bad member 000001F1B1B97E41 at 000001F1B1BC0118
Last good object: 000001F1B1BC00E8.
object 000001f1b1bc0340: bad member 000001F1B1BC0118 at 000001F1B1BC0348
Last good object: 000001F1B1BC0318.
The corrupted addresses are very near valid addresses:
!do 000001f1b1bc0340
Name: Voron.Util.PageTable+PagesBuffer
MethodTable: 00007ffbf4091728
EEClass: 00007ffbf408dcd0
Size: 32(0x20) bytes
File: C:\Work\ravendb4\test\Tryouts\bin\Debug\netcoreapp2.2 - Copy\Voron.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffbf40b8008 40004ab 8 ...pl.PagePosition[] 0 instance 000001f1b1bc0118 PagePositions
00007ffc45156a70 40004ac 10 System.Int32 1 instance 0 Start
00007ffc45156a70 40004ad 14 System.Int32 1 instance 1 End
0:077> !DumpObj /d 000001f1b1bc0118
<Note: this object has an invalid CLASS field>
Invalid object
I suspect this perhaps should be 000001f1b1bc0108
( vs ...18
) which means perhaps one bit flipped in the address value:
0:077> !DumpArray /d 000001f1b1bc0108
Name: Voron.Impl.PagePosition[]
MethodTable: 00007ffbf40b8008
EEClass: 00007ffc447f6c08
Size: 40(0x28) bytes
Array: Rank 1, Number of elements 2, Type CLASS
Element Methodtable: 00007ffbf407edb0
[0] 000001f1b1b97e41
[1] null
array element at 0 is also corrupt, looks like here perhaps the low bit got flipped:
0:077> !DumpObj /d 000001f1b1b97e41
<Note: this object has an invalid CLASS field>
Invalid object
0:077> !DumpObj /d 000001f1b1b97e40
Name: Voron.Impl.PagePosition
MethodTable: 00007ffbf407edb0
EEClass: 00007ffbf408cac0
Size: 48(0x30) bytes
File: C:\Work\ravendb4\test\Tryouts\bin\Debug\netcoreapp2.2 - Copy\Voron.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffc45156b70 40001dd 8 System.Int64 1 instance 2 TransactionId
00007ffc45156b70 40001de 10 System.Int64 1 instance 1 JournalNumber
00007ffc45156b70 40001df 18 System.Int64 1 instance 414 ScratchPage
00007ffc45156a70 40001e0 20 System.Int32 1 instance 0 ScratchNumber
00007ffc45154bb0 40001e1 24 System.Boolean 1 instance 0 IsFreedPageMarker
00007ffc45154bb0 40001e2 25 System.Boolean 1 instance 0 UnusedInPTT
My guess would be that perhaps something in the app is corrupting these values. These single bit flips do not look like the pattern we'd see if we had an issue in GC reporting.
Does this app do single-bit updates to memory (say via pointers)?
Dump 1 looks similar -- same stack and a bit is cleared in a method table address for an object in an array.
Dump 2 seems to have more severe corruption and a different stack.
@myarichuk Could I trouble you to re-run this test with stress logs enabled? This would be "set COMPlus_StressLog=1" as an environment variable before running the test and reproducing the crash. (The logs themselves are captured inside the crash dump, so you only have to share the dump...no extra files.)
This greatly helps us track down this kind of failure. Thank you!
Thanks for looking into this! Sure, I will rerun this with this env variable and share the results asap.
Does this app do single-bit updates to memory (say via pointers)?
There are couple of places we do single-bit flips through pointers, but those are heavily tested areas and judging by the corrupted objects pretty far from those code areas.
The "single bit flipping" sounded suspicious to me so I checked my memory with memtest86 and tried to reproduce this issue on another machine. It appears this was happening due to hardware issue in memory sticks because: A) it didn't reproduce on another machine B) memtest86 has shown I had hardware issues in my memory
Thus, I am closing this issue because I think the underlying reason for this access violation exception is faulty memory sticks. What do you think? Can there be any other issue besides faulty memory?
What do you think? Can there be any other issue besides faulty memory?
It's very likely this is due to faulty memory. Even if it's not, the very first step will be to reproduce this issue on a machine where the memory doesn't fault. =)
Please do feel free to file another issue if you see this again on another machine!
The exception happens intermittently when running specific RavenDB test in a 64-bit Windows process.
dotnet --info
returns the following:The following code was used to reproduce this. On my machine reproduction takes between 30 minutes to several hours to happen (the code for the test itself can be found here: https://github.com/ravendb/ravendb/blob/v4.2/test/SlowTests/Server/Documents/PeriodicBackup/RavenDB-11139.cs)
The stack trace of the AccessViolationException typically looks like this:
I have captured several dumps from when the AVE happened. They are available to download from this link: https://drive.google.com/file/d/1bNY6GZw3BhpsK73i8naiEB-AY_-LdNjy/view?usp=sharing