AccessViolation on the gc phase

[GeorgeAlexandria]

Hello,

AccessViolation (AV) was raised a couple of time on the our app under .net472 at a different state of GC, but unfortunately I can't attach dump of app. So I wrote a simple app (AccessViolation.zip) that try to emulate work with memory as is doing our app. Simple app was running in 4-6 instances in parallel on the 16gb of RAM under 64bit preferred in Debug configuration under Win10 and a couple of dump was created when AV was raised:

1) Dump: https://drive.google.com/file/d/10DcihB-Tom_kMGGIUtLolo2THMD_lDl2/view with AV on clr!WKS::gc_heap::mark_object_simple

Windbg kb output

``` 0:009> kb # RetAddr : Args to Child : Call Site 00 00007ffa`0d49d051 : 0000020d`70de0000 0000020d`71e11c30 000000d9`5fafeac0 00007ffa`0d4947a0 : clr!WKS::gc_heap::mark_object_simple+0x113 01 00007ffa`0d4952ba : 00007ffa`0d4947a0 00000000`00000000 00000000`00000000 00000000`00000000 : clr!WKS::gc_heap::mark_through_cards_for_large_objects+0x36f 02 00007ffa`0d495d3f : 0000020d`117ffa01 00000000`00000000 00000000`00000000 00000000`00000001 : clr!WKS::gc_heap::mark_phase+0x23a 03 00007ffa`0d495c53 : 0000020d`117ffa01 00000000`00000001 00000000`00000000 00000000`00000000 : clr!WKS::gc_heap::gc1+0xa3 04 00007ffa`0d497a97 : 0000020d`117ffa01 00000000`00000000 00000000`00000000 00000000`00000000 : clr!WKS::gc_heap::garbage_collect+0x193 05 00007ffa`0d499ce7 : 0000020d`117ffad0 00000000`00000000 00000000`00000018 0000020d`781d3898 : clr!WKS::GCHeap::GarbageCollectGeneration+0xef 06 00007ffa`0d43c7b3 : 00007ffa`0d492550 000000d9`5faff0c0 0000020d`781d3898 00000000`00000000 : clr!WKS::GCHeap::Alloc+0x29c 07 00007ff9`adef0b9d : 00007ff9`adde5af0 0000020d`5e1c2e28 0000020d`5e6d8040 00000000`00000000 : clr!JIT_New+0x339 08 00007ffa`0c3bdf12 : 0000020d`5e1c2e28 00007ffa`0d5aa715 00007ffa`0d43b5b5 00007ffa`42d34a91 : 0x00007ff9`adef0b9d 09 00007ffa`0c3bdd95 : 0000020d`78569618 00007ffa`217f2874 0000020d`56641000 00000000`00000018 : mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003AEF+0x172 0a 00007ffa`0c3bdd65 : 0000020d`5e1c3158 00007ffa`217f4173 00000000`00001001 0000020d`56641000 : mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003AEE+0x15 0b 00007ffa`0c463e85 : 000000d9`5faff378 000000d9`5faff518 00007ffa`0be31000 00007ffa`0d437255 : mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)$##6003AED+0x55 0c 00007ffa`0d436913 : 0000020d`5e1c3180 00007ffa`0bed4430 00007ffa`0c01c4b8 00000000`00000000 : mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()$##6003C00+0x55 0d 00007ffa`0d436818 : 000000d9`5faff538 00007ffa`0d4373bc 00000000`00000000 00007ffa`0d437326 : clr!CallDescrWorkerInternal+0x83 0e 00007ffa`0d4370d8 : 000000d9`5faff538 000000d9`5faff640 000000d9`5faff470 00000000`00000001 : clr!CallDescrWorkerWithHandler+0x4e 0f 00007ffa`0d5aa85f : 000000d9`5faff8d0 00000000`00000001 00007ffa`0bed4430 00000000`00000001 : clr!MethodDescCallSite::CallTargetWorker+0x102 10 00007ffa`0d437cd0 : 0000020d`5e1c3180 00007ffa`0d5ab830 000000d9`5faff810 00007ffa`0d43db37 : clr!ThreadNative::KickOffThread_Worker+0xffffffff`fffff02f 11 00007ffa`0d437c43 : 000000d9`5faff810 0000020d`781d3830 00000000`00000000 00007ffa`0d435341 : clr!ManagedThreadBase_DispatchInner+0x40 12 00007ffa`0d437b82 : 000000d9`5faff810 00000000`00000000 00007ffa`42cd0000 00007ffa`42d456c3 : clr!ManagedThreadBase_DispatchMiddle+0x6c 13 00007ffa`0d437d67 : ffffffff`ffffffff 0000020d`781d3830 000000d9`5faff7d0 0000020d`57266fd0 : clr!ManagedThreadBase_DispatchOuter+0x4c 14 00007ffa`0d5aa749 : 0000020d`781d3830 000000d9`00000001 00000000`00000001 00000000`00000003 : clr!ManagedThreadBase_FullTransitionWithAD+0x2f 15 00007ffa`0d43b5b5 : 0000020d`7830efe0 0000ffff`00000001 0000020d`781d3830 000000d9`5faff8a8 : clr!ThreadNative::KickOffThread+0xe6 16 00007ffa`407b7974 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : clr!Thread::intermediateThreadProc+0x8b 17 00007ffa`42d3a271 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14 18 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21 ```

2) Dump: https://drive.google.com/file/d/1Mr3jBGDdbZkBLZzd7C-fRhDkZtbf7y9l/view with AV on clr!WKS::gc_heap::background_mark_simple1

Windbg kb output

``` 0:010> kb # RetAddr : Args to Child : Call Site 00 00007ffd`fe627a9c : 00000000`00000020 0000025a`5de57000 00000000`0000004f 00007ffd`fe48c0fe : clr!WKS::gc_heap::background_mark_simple1+0x23c 01 00007ffd`fe628863 : 00000008`00000002 00007ffd`fe627de0 00000000`00000003 00007ffd`fe62a7f0 : clr!WKS::gc_heap::background_mark_simple+0x6d 02 00007ffd`fe627ceb : ffffffff`ffffffff 00007ffd`fe627de0 00007ffd`fe62a7f0 00007ffd`fec2757c : clr!WKS::gc_heap::background_drain_mark_list+0x38 03 00007ffd`fe627441 : 000000d2`9f3fe201 00007ffd`fe62a7f0 00000000`00000002 00007ffd`fe489759 : clr!WKS::gc_heap::background_mark_phase+0x24c 04 00007ffd`fe6273bd : 000000d2`9f3fe201 0000025a`7ed23830 00007ffd`fe62a7f0 00000000`00000000 : clr!WKS::gc_heap::gc1+0x1817a5 05 00007ffd`fe7742d5 : 000000d2`9f3fe201 0000025a`7ed23830 00007ffd`fe62a7f0 000000d2`9f3fe2c0 : clr!WKS::gc_heap::bgc_thread_function+0xf0 06 00007ffd`fe44b5b5 : 000000d2`9f3fe2c0 000000d2`9f6ff7d0 00000000`00000000 00007ffd`fe774250 : clr!::operator()+0x7d 07 00007ffe`1ec37974 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : clr!Thread::intermediateThreadProc+0x8b 08 00007ffe`1edaa271 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14 09 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21 ```

3) Dump: https://drive.google.com/file/d/1RRGrMzxIJdnUqyrl20KT5YG8NhWPwppo/view with AV on clr!WKS::gc_heap::mark_object_simple

Windbg kb output

``` 0:010> kb # RetAddr : Args to Child : Call Site 00 00007ffa`0d49d051 : 0000018b`6896f600 0000018b`68986840 00000021`5aafe7f0 00007ffa`0d4947a0 : clr!WKS::gc_heap::mark_object_simple+0x72 01 00007ffa`0d4952ba : 00007ffa`0d4947a0 00000000`00000000 00000000`00000000 00000000`00000000 : clr!WKS::gc_heap::mark_through_cards_for_large_objects+0x36f 02 00007ffa`0d495d3f : 0000018b`229fea01 00000000`00000000 00000000`00000000 00000000`00000001 : clr!WKS::gc_heap::mark_phase+0x23a 03 00007ffa`0d495c53 : 0000018b`229fea01 00000000`00000001 00000000`00000000 00000000`00000000 : clr!WKS::gc_heap::gc1+0xa3 04 00007ffa`0d497a97 : 0000018b`229fea01 00000000`00000000 00000000`00000000 00000000`00000000 : clr!WKS::gc_heap::garbage_collect+0x193 05 00007ffa`0d499ce7 : 0000018b`229fea30 00000000`00000000 00000000`00000018 0000018b`6ed2b898 : clr!WKS::GCHeap::GarbageCollectGeneration+0xef 06 00007ffa`0d43c7b3 : 00007ffa`0d492550 00000021`5aafedf0 0000018b`6ed2b898 0000018b`10de6d50 : clr!WKS::GCHeap::Alloc+0x29c 07 00007ff9`adf00b9d : 00007ff9`addf5af0 0000018b`54d22e28 0000018b`54d26140 00000000`00000000 : clr!JIT_New+0x339 08 00007ffa`0c3bdf12 : 0000018b`54d22e28 00007ffa`0d5aa715 00007ffa`0d43b5b5 00007ffa`42d34a91 : 0x00007ff9`adf00b9d 09 00007ffa`0c3bdd95 : 0000018b`6f0c96e8 00007ffa`217f2874 0000018b`4d151000 00000000`00000018 : mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003AEF+0x172 0a 00007ffa`0c3bdd65 : 0000018b`54d23158 00007ffa`217f4173 00000000`00001001 0000018b`4d151000 : mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003AEE+0x15 0b 00007ffa`0c463e85 : 00000021`5aaff0a8 00000021`5aaff248 00007ffa`0be31000 00007ffa`0d437255 : mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)$##6003AED+0x55 0c 00007ffa`0d436913 : 0000018b`54d23180 00007ffa`0bed4430 00007ffa`0c01c4b8 00000000`00000000 : mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()$##6003C00+0x55 0d 00007ffa`0d436818 : 00000021`5aaff268 00007ffa`0d4373bc 00000000`00000000 00007ffa`0d437326 : clr!CallDescrWorkerInternal+0x83 0e 00007ffa`0d4370d8 : 00000021`5aaff268 00000021`5aaff370 00000021`5aaff1a0 00000000`00000001 : clr!CallDescrWorkerWithHandler+0x4e 0f 00007ffa`0d5aa85f : 00000021`5aaff600 00000000`00000001 00007ffa`0bed4430 00000000`00000001 : clr!MethodDescCallSite::CallTargetWorker+0x102 10 00007ffa`0d437cd0 : 0000018b`54d23180 00007ffa`0d5ab830 00000021`5aaff540 00007ffa`0d43db37 : clr!ThreadNative::KickOffThread_Worker+0xffffffff`fffff02f 11 00007ffa`0d437c43 : 00000021`5aaff540 0000018b`6ed2b830 00000000`00000000 00007ffa`0d435341 : clr!ManagedThreadBase_DispatchInner+0x40 12 00007ffa`0d437b82 : 00000021`5aaff540 00000000`00000000 00007ffa`42cd0000 00007ffa`42d456c3 : clr!ManagedThreadBase_DispatchMiddle+0x6c 13 00007ffa`0d437d67 : ffffffff`ffffffff 0000018b`6ed2b830 00000021`5aaff500 0000018b`4dd6efd0 : clr!ManagedThreadBase_DispatchOuter+0x4c 14 00007ffa`0d5aa749 : 0000018b`6ed2b830 00000021`00000001 00000000`00000001 00000000`00000003 : clr!ManagedThreadBase_FullTransitionWithAD+0x2f 15 00007ffa`0d43b5b5 : 0000018b`6ee64fe0 0000ffff`00000001 0000018b`6ed2b830 00000021`5aaff5d8 : clr!ThreadNative::KickOffThread+0xe6 16 00007ffa`407b7974 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : clr!Thread::intermediateThreadProc+0x8b 17 00007ffa`42d3a271 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14 18 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21 ```

4) Dump: https://drive.google.com/file/d/1-Em0oKOPf_EMg23M45koIriR8H5fAtIU/view with AV on clr!WKS::gc_heap::mark_object_simple

Windbg kb output

``` 0:009> kb # RetAddr : Args to Child : Call Site 00 00007ffa`0d49d051 : 000001a5`90eafa00 000001a5`90ec0590 0000000f`85afe730 00007ffa`0d4947a0 : clr!WKS::gc_heap::mark_object_simple+0x1ad 01 00007ffa`0d4952ba : 00007ffa`0d4947a0 00000000`00000000 000001a6`00000000 00000000`00000000 : clr!WKS::gc_heap::mark_through_cards_for_large_objects+0x36f 02 00007ffa`0d495d3f : 000001a6`36fcde01 00000000`00000000 00000000`00000000 00000000`00000001 : clr!WKS::gc_heap::mark_phase+0x23a 03 00007ffa`0d495c53 : 000001a6`36fcde01 00000000`00000001 00000000`00000000 00000000`00000000 : clr!WKS::gc_heap::gc1+0xa3 04 00007ffa`0d497a97 : 000001a6`36fcde01 00000000`00000000 00000000`00000000 00000000`00000000 : clr!WKS::gc_heap::garbage_collect+0x193 05 00007ffa`0d499ce7 : 000001a6`36fcdea8 00000000`00000000 00000000`00000018 000001a5`f0067898 : clr!WKS::GCHeap::GarbageCollectGeneration+0xef 06 00007ffa`0d43c7b3 : 00007ffa`0d492550 0000000f`85afed30 000001a5`f0067898 000001a5`a81c7448 : clr!WKS::GCHeap::Alloc+0x29c 07 00007ff9`adf00bd5 : 00007ff9`addf5af0 000001a5`80002e28 000001a5`80003328 00000000`00000000 : clr!JIT_New+0x339 08 00007ffa`0c3bdf12 : 000001a5`800030b8 000001a5`800030b8 0000000f`85afed70 00000000`00000000 : 0x00007ff9`adf00bd5 09 00007ffa`0c3bdd95 : 000001a5`f03f9c30 00007ffa`217f2874 000001a5`e6361000 00000000`00000018 : mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003AEF+0x172 0a 00007ffa`0c3bdd65 : 000001a5`80003158 00007ffa`217f4173 00000000`00001001 000001a5`e6361000 : mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003AEE+0x15 0b 00007ffa`0c463e85 : 0000000f`85afefe8 0000000f`85aff188 00007ffa`0be31000 00007ffa`0d437255 : mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)$##6003AED+0x55 0c 00007ffa`0d436913 : 000001a5`80003180 00007ffa`0bed4430 00007ffa`0c01c4b8 00000000`00000000 : mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()$##6003C00+0x55 0d 00007ffa`0d436818 : 0000000f`85aff1a8 00007ffa`0d4373bc 00000000`00000000 00007ffa`0d437326 : clr!CallDescrWorkerInternal+0x83 0e 00007ffa`0d4370d8 : 0000000f`85aff1a8 0000000f`85aff2b0 0000000f`85aff0e0 00000000`00000001 : clr!CallDescrWorkerWithHandler+0x4e 0f 00007ffa`0d5aa85f : 0000000f`85aff540 00000000`00000001 00007ffa`0bed4430 00000000`00000001 : clr!MethodDescCallSite::CallTargetWorker+0x102 10 00007ffa`0d437cd0 : 000001a5`80003180 00007ffa`0d5ab830 0000000f`85aff480 00007ffa`0d43db37 : clr!ThreadNative::KickOffThread_Worker+0xffffffff`fffff02f 11 00007ffa`0d437c43 : 0000000f`85aff480 00007ffa`0d434090 00000000`00000000 00007ffa`0d435341 : clr!ManagedThreadBase_DispatchInner+0x40 12 00007ffa`0d437b82 : 0000000f`85aff480 00000000`00000000 00000000`00000000 00000000`00000000 : clr!ManagedThreadBase_DispatchMiddle+0x6c 13 00007ffa`0d437d67 : ffffffff`ffffffff 000001a5`f0067830 0000000f`85aff440 000001a5`e6f8afd0 : clr!ManagedThreadBase_DispatchOuter+0x4c 14 00007ffa`0d5aa749 : 000001a5`f0067830 0000000f`00000001 00000000`00000001 00000000`00000003 : clr!ManagedThreadBase_FullTransitionWithAD+0x2f 15 00007ffa`0d43b5b5 : 000001a5`f0150fe0 00000000`00000001 000001a5`f0067830 0000000f`85aff518 : clr!ThreadNative::KickOffThread+0xe6 16 00007ffa`407b7974 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : clr!Thread::intermediateThreadProc+0x8b 17 00007ffa`42d3a271 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14 18 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21 ```

5) Dump: https://drive.google.com/file/d/1Z-p4ZYNIVe3CT9AIFvmwFR6YLPkl1xcr/view with AV on clr!WKS::gc_heap::mark_object_simple1

Windbg kb output

``` 0:006> kb # RetAddr : Args to Child : Call Site 00 00007ffa`0d497ce6 : 00000207`a0344c28 00000207`a0344c18 00000000`00000000 00000207`1fe8f000 : clr!WKS::gc_heap::mark_object_simple1+0x220 01 00007ffa`0d49d051 : 00000207`3aa26a00 00000207`3aad55d0 00000039`c73feac0 00007ffa`0d4947a0 : clr!WKS::gc_heap::mark_object_simple+0x1e0 02 00007ffa`0d4952ba : 00007ffa`0d4947a0 00000000`00000000 00000000`00000000 00000000`00000000 : clr!WKS::gc_heap::mark_through_cards_for_large_objects+0x36f 03 00007ffa`0d495d3f : 00000207`a0371b01 00000000`00000000 00000000`00000000 00000000`00000000 : clr!WKS::gc_heap::mark_phase+0x23a 04 00007ffa`0d495c53 : 00000207`a0371b01 00000000`00000001 00000000`00000000 00000000`00000000 : clr!WKS::gc_heap::gc1+0xa3 05 00007ffa`0d497a97 : 00000207`a0371b01 00000000`00000000 00000000`00000000 00000000`00000000 : clr!WKS::gc_heap::garbage_collect+0x193 06 00007ffa`0d499ce7 : 00000207`a0371bd8 00000000`00000000 00000000`00000018 00000207`40ddb898 : clr!WKS::GCHeap::GarbageCollectGeneration+0xef 07 00007ffa`0d43c7b3 : 00007ffa`0d492550 00000039`c73ff0c0 00000207`40ddb898 00000207`26e11de8 : clr!WKS::GCHeap::Alloc+0x29c 08 00007ff9`adf10b9d : 00007ff9`ade05af0 00000207`26dd2978 00000207`26de0128 00000000`00000000 : clr!JIT_New+0x339 09 00007ffa`0c3bdf12 : 00000207`26dd2e28 00007ffa`0d5aa715 00007ffa`0d43b5b5 00007ffa`42d34a91 : 0x00007ff9`adf10b9d 0a 00007ffa`0c3bdd95 : 00000207`41179e38 00007ffa`217f2874 00000207`1ef71000 00000000`00000018 : mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003AEF+0x172 0b 00007ffa`0c3bdd65 : 00000207`26dd3050 00007ffa`217f4173 00000000`00001001 00000207`1ef71000 : mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003AEE+0x15 0c 00007ffa`0c463e85 : 00000039`c73ff378 00000039`c73ff518 00007ffa`0be31000 00007ffa`0d437255 : mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)$##6003AED+0x55 0d 00007ffa`0d436913 : 00000207`26dd3078 00007ffa`0bed4430 00007ffa`0c01c4b8 00000000`00000000 : mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()$##6003C00+0x55 0e 00007ffa`0d436818 : 00000039`c73ff538 00007ffa`0d4373bc 00000000`00000000 00007ffa`0d437326 : clr!CallDescrWorkerInternal+0x83 0f 00007ffa`0d4370d8 : 00000039`c73ff538 00000039`c73ff640 00000039`c73ff470 00000000`00000001 : clr!CallDescrWorkerWithHandler+0x4e 10 00007ffa`0d5aa85f : 00000039`c73ff8d0 00000000`00000001 00007ffa`0bed4430 00000000`00000001 : clr!MethodDescCallSite::CallTargetWorker+0x102 11 00007ffa`0d437cd0 : 00000207`26dd3078 00007ffa`0d5ab830 00000039`c73ff810 00007ffa`0d43db37 : clr!ThreadNative::KickOffThread_Worker+0xffffffff`fffff02f 12 00007ffa`0d437c43 : 00000039`c73ff810 00000207`40ddb830 00000000`00000000 00007ffa`0d435341 : clr!ManagedThreadBase_DispatchInner+0x40 13 00007ffa`0d437b82 : 00000039`c73ff810 00000000`00000000 00000000`00000000 00000000`00000000 : clr!ManagedThreadBase_DispatchMiddle+0x6c 14 00007ffa`0d437d67 : ffffffff`ffffffff 00000207`40ddb830 00000039`c73ff7d0 00000207`1fb96fd0 : clr!ManagedThreadBase_DispatchOuter+0x4c 15 00007ffa`0d5aa749 : 00000207`40ddb830 00000039`00000001 00000000`00000001 00000000`00000003 : clr!ManagedThreadBase_FullTransitionWithAD+0x2f 16 00007ffa`0d43b5b5 : 00000207`40ef6fe0 00000000`00000001 00000207`40ddb830 00000039`c73ff8a8 : clr!ThreadNative::KickOffThread+0xe6 17 00007ffa`407b7974 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : clr!Thread::intermediateThreadProc+0x8b 18 00007ffa`42d3a271 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14 19 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21 ```

Any suggestions would be grateful.

[ghost]

Tagging subscribers to this area: @dotnet/gc See info in area-owners.md if you want to be subscribed.

[alexxn]

Upvoting this bug.

[danmoseley]

Does it repro on.NET 5.0 preview? That might be helpful to know.

[GeorgeAlexandria]

@danmosemsft, currently didn't try to repro it on .NET 5 preview or some actual version of .NET Core. Can try to repro if it's necessary, but it may take a time

[Maoni0]

I have been running this (16 instances) on one of my machines for the past 8 hours with the current 5.0 build with no repro.

@ChrisAhna is this something you have cycles to help with? thanks so much!

[GeorgeAlexandria]

After installation .NET 5 preview 7 sdk and building app under .net5.0 by visual studio 16.7.1 AV still persists:

1) Dump: https://drive.google.com/file/d/1yCk1lCsRYA2fx52vqP1gvzUQMbby4O0y/view with AV on coreclr!MethodTable::GetFlag [inlined in coreclr!WKS::gc_heap::background_mark_simple1]

2) Dump: https://drive.google.com/file/d/15ODUo_sIoAVaZm9SkyTezeLScxGBt-7W/view with AV on coreclr!MethodTable::GetFlag [inlined in coreclr!WKS::gc_heap::background_sweep]

3) Dump: https://drive.google.com/file/d/17d6Its8aHOjg4TcamyE_GSy5Ocs1oqUV/view with AV on coreclr!MethodTable::GetFlag [inlined in coreclr!WKS::gc_heap::mark_object_simple]

4) Dump: https://drive.google.com/file/d/1tM6GpFxYPZ5HfCP2izxaALi1Zfe3ud6K/view with AV on coreclr!MethodTable::GetFlag [inlined in coreclr!WKS::gc_heap::plan_phase]

5) Dump: https://drive.google.com/file/d/1nL3QnkYnbblNUx46Lc-rAx6xeBs6jO2Q/view with AV on coreclr!MethodTable::GetFlag [inlined in coreclr!WKS::gc_heap::mark_object_simple1]

By the way, seems that AV is raised more often on .net5 than on .net472 : took, presumably, 10 dumps in a few hours on .net5 versus 6-7 dumps in a few days on .net472.

[mangod9]

Thanks @GeorgeAlexandria to trying on .net 5. We will investigate.

[mangod9]

possible heap corruption:

0:008> !dumpheap -stat
Could not request method table data for object 000001E80016D848 (MethodTable: 00007FF816640068).

[cshung]

Could it be https://github.com/dotnet/runtime/pull/38784?

[Maoni0]

that only affects running in containers and I didn't see container mentioned for this issue (and in test there's no forced LOH compaction). maybe I missed reading something?

[cshung]

that only affects running in containers and I didn't see container mentioned for this issue (and in test there's no forced LOH compaction). maybe I missed reading something?

Heap corruption, AV on background_mark_object trying to access method table. All seems to look like what I was investigating, so I just pattern-matched it. I didn't really look into it in depth.

[GeorgeAlexandria]

It was not run in containers and app didn't forced LOH compaction or GC

[ChrisAhna]

Thanks for this report. There is probably something about the underlying issue that makes some machines much more likely to see a crash compared to others. I have been fanning out the repro (on both net472 and net50) for the past few days across a few different execution patterns (i.e., varying parallelism and timeout), and haven't seen any crashes yet.

I'm going to spend more time digging in the dump files to see if there are any clues about what went wrong when the crashes did occur.

[ChrisAhna]

Looking at the 10 dumps, 8 of them show that data has been corrupted in one of the following ways:

• 5 cases where bit 0x00400000 is unexpectedly set at offset 0x848 within the enclosing 4KB page.
• 3 cases where bit 0x00004000 is unexpectedly clear at offset 0xc30 within the enclosing 4KB page.

This could certainly be caused by some obscure software problem. That said, seeing a consistent page offset (i.e., low 12 bits of the address) across the corruption sites suggests that the problem may be happening at a lower level. For example, while a given piece of physical RAM can be accessed through any number of virtual addresses, the low 12 bits will be the same across all of them. So if one piece of RAM was prone to occasional bit flips, the low 12 virtual address bits would be the same across all of the places where this bit flip ends up occurring during program execution.

From what I can see in the dump files, it looks like all 10 crashes occurred on the same physical machine.

Have you seen similar crashes on other physical machines? If so, is it possible to share a few crash dumps collected in those cases? Thanks.

For reference:

The 5 cases where bit 0x00400000 is unexpectedly set at offset 0x848 within the enclosing 4KB page are:

net5: https://drive.google.com/file/d/1yCk1lCsRYA2fx52vqP1gvzUQMbby4O0y/view 000001664D605848 holds 00007FF816620068; 00007FF816220068 would have been correct

net5: https://drive.google.com/file/d/15ODUo_sIoAVaZm9SkyTezeLScxGBt-7W/view 000001E80016D848 holds 00007FF816640068; 00007FF816240068 would have been correct

net5: https://drive.google.com/file/d/17d6Its8aHOjg4TcamyE_GSy5Ocs1oqUV/view 000001B57FD9F848 holds 00007FF816630068; 00007FF816230068 would have been correct

net5: https://drive.google.com/file/d/1nL3QnkYnbblNUx46Lc-rAx6xeBs6jO2Q/view 000002CE4B1E9848 holds 00007FF816620068; 00007FF816220068 would have been correct

net472: https://drive.google.com/file/d/1RRGrMzxIJdnUqyrl20KT5YG8NhWPwppo/view 0000018b68986848 holds 0000018b2177ca88; 0000018b2137ca88 would have been correct

The 3 cases where bit 0x00004000 is unexpectedly clear at offset 0xc30 within the enclosing 4KB page are:

net472: https://drive.google.com/file/d/10DcihB-Tom_kMGGIUtLolo2THMD_lDl2/view 0000020d71e11c30 holds 0000020d115c9130; 0000020d115cd130 would have been correct

net472: https://drive.google.com/file/d/1Mr3jBGDdbZkBLZzd7C-fRhDkZtbf7y9l/view 0000025a42ddec30 holds 0000025a42ddac40; 0000025a42ddec40 would have been correct

net472: https://drive.google.com/file/d/1Z-p4ZYNIVe3CT9AIFvmwFR6YLPkl1xcr/view 00000207a0344c30 holds 00000207a0340c40; 00000207a0344c40 would have been correct

[mangod9]

Thanks @ChrisAhna for investigating. Moving to 6.0 based on the current theory that this is somehow hardware related.

[danmoseley]

Would it help if @GeorgeAlexandria ran some memory tester tool, often the firmware includes one.

[mangod9]

yeah that would be helpful, or to try a repro on a different physical machine.

[GeorgeAlexandria]

Tried to repro it on the different physical machines at the last several days, but have not seen any crashes. So will run memory tool to check and will continue trying to repro. Any preferences in memory tools?

[jkotas]

I would start with the one that comes with Windows (look for Windows Memory Diagnostic app).

[mangod9]

Hey @GeorgeAlexandria any update on this? If this isnt reproable on separate hardware perhaps we can close this. Thx!

[GeorgeAlexandria]

Have not seen any crashes on other different machines and found a few problems with memory on the first machine. So I decided to replace memory on it and try to repro crashes again, after that I will be sure to close issue or not. (Unfortunately, need additional time to replace RAM on the first machine)

[mangod9]

ok, sounds good.

[GeorgeAlexandria]

After replacing memory and testing within a several days crashes didn't reproduce anymore. So currently I will close the issue, but if it will raise again in a future – will reopen issue or create a new one.

[mangod9]

Thanks for the follow up.