search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.97k stars 4.66k forks source link

Process crashed with Fatal error. Internal CLR error. (0x80131506) #89165

Open karmeli87 opened 1 year ago

karmeli87 commented 1 year ago

Description

We are running some stress tests on our database by inserting hundred of millions of documents. During one such test we got the following crash:

Fatal error. Internal CLR error. (0x80131506)
   at System.Runtime.CompilerServices.CastHelpers.StelemRef_Helper_NoCacheLookup(System.Object ByRef, Void*, System.Object)
   at System.Runtime.CompilerServices.CastHelpers.StelemRef_Helper(System.Object ByRef, Void*, System.Object)
   at Sparrow.Json.CachedProperties.UnlikelySortProperties(Sparrow.Collections.FastList`1<PropertyTag>)
   at Sparrow.Json.BlittableWriter`1[[Sparrow.Json.UnmanagedWriteBuffer, Sparrow, Version=6.0.0.0, Culture=neutral, PublicKeyToken=37f41c7f99471593]].WriteObjectMetadata(Sparrow.Collections.FastList`1<PropertyTag>, Int64, Int32)
   at Sparrow.Json.BlittableJsonDocumentBuilder.ReadInternal[[Sparrow.Json.BlittableJsonDocumentBuilder+WriteNone, Sparrow, Version=6.0.0.0, Culture=neutral, PublicKeyToken=37f41c7f99471593]]()
   at Sparrow.Json.JsonOperationContext+<ParseToMemoryAsync>d__65.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Sparrow.Json.JsonOperationContext+<ParseToMemoryAsync>d__65, Sparrow, Version=6.0.0.0, Culture=neutral, PublicKeyToken=37f41c7f99471593]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at Sparrow.Utils.ZstdStream+<ReadAsync>d__35.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Sparrow.Utils.ZstdStream+<ReadAsync>d__35, Sparrow, Version=6.0.0.0, Culture=neutral, PublicKeyToken=37f41c7f99471593]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32)
   at System.Threading.Tasks.ValueTask`1+ValueTaskSourceAsTask+<>c[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<.cctor>b__4_0(System.Object)
   at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean)
   at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs)
   at System.Net.Sockets.SocketAsyncEventArgs+<>c.<.cctor>b__176_0(UInt32, UInt32, System.Threading.NativeOverlapped*)
   at System.Threading.ThreadPoolTypedWorkItemQueue`2[[System.Threading.PortableThreadPool+IOCompletionPoller+Event, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Threading.PortableThreadPool+IOCompletionPoller+Callback, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

A dump was automatically generated and is available here: https://drive.google.com/file/d/1mXzp3mASCkLaHmxG8fojnn9oV-IYZER_/view?usp=sharing

Here is the version used: https://hibernatingrhinos.com/downloads/RavenDB%20for%20Windows%20x64/20230715-0102-Nightly

Reference to the code: https://github.com/ravendb/ravendb/blob/44a806981b4a4398b3b42bcf9604ac83d4df9d1d/src/Sparrow/Json/CachedProperties.cs#L324

OS version: Windows 10 Version 19045

CLR version: 7.0.923.32018

0:047> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

Failed to request MethodData, not in JIT code range

KEY_VALUES_STRING: 1

    Key  : AV.Dereference
    Value: NullPtr

    Key  : AV.Fault
    Value: Read

    Key  : Analysis.CPU.mSec
    Value: 2093

    Key  : Analysis.Elapsed.mSec
    Value: 3311

    Key  : Analysis.IO.Other.Mb
    Value: 0

    Key  : Analysis.IO.Read.Mb
    Value: 1

    Key  : Analysis.IO.Write.Mb
    Value: 0

    Key  : Analysis.Init.CPU.mSec
    Value: 296

    Key  : Analysis.Init.Elapsed.mSec
    Value: 2339

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 177

    Key  : CLR.Engine
    Value: CORECLR

    Key  : CLR.Version
    Value: 7.0.923.32018

    Key  : Failure.Bucket
    Value: NULL_POINTER_READ_c0000005_coreclr.dll!ObjIsInstanceOfCore

    Key  : Failure.Hash
    Value: {065d61d5-ebc8-eed8-03df-c09bfba3540a}

    Key  : Failure.Source.FileLine
    Value: 2097

    Key  : Failure.Source.FilePath
    Value: D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp

    Key  : Failure.Source.SourceServerCommand
    Value: raw.githubusercontent.com/dotnet/runtime/8e9a17b2216f51a5788f8b1c467a4cf3b769e7d7/src/coreclr/vm/jithelpers.cpp

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 382147

    Key  : Timeline.Process.Start.DeltaSec
    Value: 91190

    Key  : WER.OS.Branch
    Value: vb_release

    Key  : WER.OS.Version
    Value: 10.0.19041.1

    Key  : WER.Process.Version
    Value: 6.0.0.60

FILE_IN_CAB:  Raven.Server.exe_230717_033345.dmp

COMMENT:  
*** "C:\Program Files\Procdump\procdump.exe" -accepteula -mp -j "c:\memdumps" 2876 600 000002B280020000
*** Just-In-Time debugger. PID: 2876 Event Handle: 600 JIT Context: .jdinfo 0x2b280020000

NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

CONTEXT:  (.ecxr)
rax=000000bdcc4be458 rbx=000000bdcc4be1f0 rcx=000002b28d36e090
rdx=0000000000000130 rsi=000000bdcc4bda90 rdi=000000bdcc4bdd00
rip=00007ff810b0de76 rsp=000000bdcc4be420 rbp=000000bdcc4be4a9
 r8=0000000000000000  r9=000002b30f800050 r10=0000000000000f57
r11=0000000000000008 r12=0000000000000000 r13=0000000000000001
r14=0000000000000000 r15=000002b285c0eb00
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010244
coreclr!MethodTable::GetFlag [inlined in coreclr!ObjIsInstanceOfCore+0x8a]:
00007ff8`10b0de76 8b03            mov     eax,dword ptr [rbx] ds:000000bd`cc4be1f0=c0000005
Resetting default scope

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00007ff810b0de76 (coreclr!MethodTable::GetFlag)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000001
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: 0000000000000000
Attempt to read from address 0000000000000000

PROCESS_NAME:  Raven.Server.exe

READ_ADDRESS:  0000000000000000 

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  0000000000000000

EXCEPTION_PARAMETER2:  0000000000000000

STACK_TEXT:  
000000bd`cc4be420 00007ff8`10b0db3a     : 00000000`00000007 00000000`00000000 000000bd`cc4be5b9 00007ff8`10bf73ed : coreclr!ObjIsInstanceOfCore+0x8a
000000bd`cc4be510 00007ff7`b3d7f106     : 00007ff7`b30b74f0 000002b2`8d36e0e8 000002b3`27d24e30 00000000`00000000 : coreclr!IsInstanceOfAny_NoCacheLookup+0x8a
000000bd`cc4be670 00007ff7`b30b74f0     : 000002b2`8d36e0e8 000002b3`27d24e30 00000000`00000000 000002b3`0f816fd0 : System_Private_CoreLib!System.Runtime.CompilerServices.CastHelpers.StelemRef_Helper_NoCacheLookup+0x16
000000bd`cc4be678 000002b2`8d36e0e8     : 000002b3`27d24e30 00000000`00000000 000002b3`0f816fd0 00007ff7`b379ff29 : 0x00007ff7`b30b74f0
000000bd`cc4be680 000002b3`27d24e30     : 00000000`00000000 000002b3`0f816fd0 00007ff7`b379ff29 000002f3`8ef329b0 : 0x000002b2`8d36e0e8
000000bd`cc4be688 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x000002b3`27d24e30

FAULTING_SOURCE_LINE:  D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp

FAULTING_SOURCE_FILE:  D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp

FAULTING_SOURCE_LINE_NUMBER:  2097

FAULTING_SOURCE_SRV_COMMAND:  https://raw.githubusercontent.com/dotnet/runtime/8e9a17b2216f51a5788f8b1c467a4cf3b769e7d7/src/coreclr/vm/jithelpers.cpp

FAULTING_SOURCE_CODE:  
No source found for 'D:\a\_work\1\s\src\coreclr\vm\methodtable.h'

SYMBOL_NAME:  coreclr!ObjIsInstanceOfCore+8a

MODULE_NAME: coreclr

IMAGE_NAME:  coreclr.dll

STACK_COMMAND:  ~47s; .ecxr ; kb

FAILURE_BUCKET_ID:  NULL_POINTER_READ_c0000005_coreclr.dll!ObjIsInstanceOfCore

OS_VERSION:  10.0.19041.1

BUILDLAB_STR:  vb_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

IMAGE_VERSION:  7.0.923.32018

FAILURE_ID_HASH:  {065d61d5-ebc8-eed8-03df-c09bfba3540a}

Followup:     MachineOwner

No apperent heap corruption

0:047> !verifyheap
10,001,159 objects verified, 0 errors.
No heap corruption detected.
Thread  47
Current frame: coreclr!ObjIsInstanceOfCore + 0x8a [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:2097]
Child-SP         RetAddr          Caller, Callee
000000BDCC4BE480 00007ff810bf7917 coreclr!SVR::gc_heap::a_fit_segment_end_p + 0x187 [D:\a\_work\1\s\src\coreclr\gc\gc.cpp:16819], calling coreclr!SVR::gc_heap::adjust_limit_clr [D:\a\_work\1\s\src\coreclr\gc\gc.cpp:15839]
000000BDCC4BE500 00007ff810b0db3a coreclr!IsInstanceOfAny_NoCacheLookup + 0x8a [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:2225], calling coreclr!ObjIsInstanceOfCore [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:2080]
000000BDCC4BE520 00007ff810bf73ed coreclr!SVR::gc_heap::allocate_soh + 0x2ad [D:\a\_work\1\s\src\coreclr\gc\gc.cpp:17083], calling coreclr!SVR::gc_heap::a_fit_segment_end_p [D:\a\_work\1\s\src\coreclr\gc\gc.cpp:16682]
000000BDCC4BE590 00007ff7b3d7f106 (MethodDesc 00007ff7b115be48 + 0x16 System.Runtime.CompilerServices.CastHelpers.StelemRef_Helper_NoCacheLookup(System.Object ByRef, Void*, System.Object)), calling 00007ff810b0dab0 (stub for System.Runtime.CompilerServices.CastHelpers.IsInstanceOfAny_NoCacheLookup(Void*, System.Object))
000000BDCC4BE628 00007ff810b0db1b coreclr!IsInstanceOfAny_NoCacheLookup + 0x6b [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:2224], calling coreclr!LazyMachStateCaptureState
000000BDCC4BE660 00007ff7b3d7f106 (MethodDesc 00007ff7b115be48 + 0x16 System.Runtime.CompilerServices.CastHelpers.StelemRef_Helper_NoCacheLookup(System.Object ByRef, Void*, System.Object)), calling 00007ff810b0dab0 (stub for System.Runtime.CompilerServices.CastHelpers.IsInstanceOfAny_NoCacheLookup(Void*, System.Object))
000000BDCC4BE690 00007ff7b379ff29 (MethodDesc 00007ff7b115be30 + 0xa9 System.Runtime.CompilerServices.CastHelpers.StelemRef_Helper(System.Object ByRef, Void*, System.Object)), calling (MethodDesc 00007ff7b115be48 + 0 System.Runtime.CompilerServices.CastHelpers.StelemRef_Helper_NoCacheLookup(System.Object ByRef, Void*, System.Object))
000000BDCC4BE6B0 00007ff810bf6bf7 coreclr!SVR::GCHeap::Alloc + 0xb7 [D:\a\_work\1\s\src\coreclr\gc\gc.cpp:46314], calling coreclr!SVR::gc_heap::allocate_more_space [D:\a\_work\1\s\src\coreclr\gc\gc.cpp:18475]
000000BDCC4BE6D0 00007ff7b3e223de (MethodDesc 00007ff7b2c5b928 + 0xde Sparrow.Json.CachedProperties.UnlikelySortProperties(Sparrow.Collections.FastList`1<PropertyTag>)), calling (MethodDesc 00007ff7b115be18 + 0 System.Runtime.CompilerServices.CastHelpers.StelemRef(System.Array, IntPtr, System.Object))
000000BDCC4BE6E0 00007ff7b3dc0ec6 (MethodDesc 00007ff7b166d488 + 0x76 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000BDCC4BE760 00007ff7b3ddb768 (MethodDesc 00007ff7b2c7b7d8 + 0xe8 Sparrow.Json.BlittableWriter`1[[Sparrow.Json.UnmanagedWriteBuffer, Sparrow]].WriteObjectMetadata(Sparrow.Collections.FastList`1<PropertyTag>, Int64, Int32)), calling (MethodDesc 00007ff7b2c5b928 + 0 Sparrow.Json.CachedProperties.UnlikelySortProperties(Sparrow.Collections.FastList`1<PropertyTag>))
000000BDCC4BE7B0 00007ff7b43503aa (MethodDesc 00007ff7b3275570 + 0x26a Sparrow.Json.Parsing.UnmanagedJsonParser.Read()), calling coreclr!JIT_MemCpy
000000BDCC4BE7C0 00007ff7b3d56e5d (MethodDesc 00007ff7b11f61f0 + 0x1dd System.Collections.Generic.Dictionary`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].FindValue(System.__Canon))
000000BDCC4BE7D0 00007ff7b3ddc83f (MethodDesc 00007ff7b2c7b8f8 + 0x33f Sparrow.Json.BlittableWriter`1[[Sparrow.Json.UnmanagedWriteBuffer, Sparrow]].WriteValue(Byte*, Int32, Sparrow.Collections.FastList`1<Int32>, Sparrow.Json.BlittableJsonToken ByRef, UsageMode, System.Nullable`1<Int32>)), calling coreclr!JIT_MemCpy
000000BDCC4BE870 00007ff7b4340d26 (MethodDesc 00007ff7b3276e68 + 0xd06 Sparrow.Json.BlittableJsonDocumentBuilder.ReadInternal[[Sparrow.Json.BlittableJsonDocumentBuilder+WriteNone, Sparrow]]()), calling (MethodDesc 00007ff7b2c7b7d8 + 0 Sparrow.Json.BlittableWriter`1[[Sparrow.Json.UnmanagedWriteBuffer, Sparrow]].WriteObjectMetadata(Sparrow.Collections.FastList`1<PropertyTag>, Int64, Int32))
000000BDCC4BE8D0 00007ff810b64fb0 coreclr!JIT_GetSharedGCThreadStaticBase + 0x70 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1830], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000BDCC4BEE60 00007ff7b4480124 (MethodDesc 00007ff7b384fe00 + 0x1404 Sparrow.Json.JsonOperationContext+<ParseToMemoryAsync>d__65.MoveNext()), calling (MethodDesc 00007ff7b3276e68 + 0 Sparrow.Json.BlittableJsonDocumentBuilder.ReadInternal[[Sparrow.Json.BlittableJsonDocumentBuilder+WriteNone, Sparrow]]())
000000BDCC4BEE90 000000005b3943d7 libzstd_win_x64!ZSTD_decompressStream_simpleArgs + 0x59b7, calling libzstd_win_x64!ZSTD_decompressStream_simpleArgs + 0xe20
000000BDCC4BF1B0 00007ff7b3dc0ec6 (MethodDesc 00007ff7b166d488 + 0x76 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000BDCC4BF220 00007ff7b4833b26 (MethodDesc 00007ff7b40e6518 + 0xf6 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Sparrow.Json.JsonOperationContext+<ParseToMemoryAsync>d__65, Sparrow]].MoveNext(System.Threading.Thread)), calling (MethodDesc 00007ff7b166d488 + 0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000BDCC4BF230 00007ff7b3e2f9a1 (MethodDesc 00007ff7b1635098 + 0x21 System.Threading.Tasks.Task.FinishContinuations()), calling 00007ff810c125d0 (stub for System.Threading.Interlocked.Exchange(System.Object ByRef, System.Object))
000000BDCC4BF260 00007ff810b64fb0 coreclr!JIT_GetSharedGCThreadStaticBase + 0x70 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1830], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000BDCC4BF2A0 00007ff7b4354197 (MethodDesc 00007ff7b1d7dc60 + 0xa7 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
000000BDCC4BF2C0 00007ff7b4374b47 (MethodDesc 00007ff7b344c200 + 0x97 Nito.Disposables.SingleDisposable`1[[System.__Canon, System.Private.CoreLib]].Dispose()), calling (MethodDesc 00007ff7b344c200 + 0xaf Nito.Disposables.SingleDisposable`1[[System.__Canon, System.Private.CoreLib]].Dispose())
000000BDCC4BF310 00007ff7b4348677 (MethodDesc 00007ff7b16350b0 + 0x77 System.Threading.Tasks.Task.RunContinuations(System.Object)), calling (MethodDesc 00007ff7b1d7dc60 + 0 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
000000BDCC4BF330 00007ff7b4534569 (MethodDesc 00007ff7b396e4b0 + 0x5a9 Sparrow.Utils.ZstdStream+<ReadAsync>d__35.MoveNext())
000000BDCC4BF3B0 00007ff7b4534495 (MethodDesc 00007ff7b396e4b0 + 0x4d5 Sparrow.Utils.ZstdStream+<ReadAsync>d__35.MoveNext()), calling (MethodDesc 00007ff7b1635098 + 0 System.Threading.Tasks.Task.FinishContinuations())
000000BDCC4BF3D0 00007ff7b435cb86 (MethodDesc 00007ff7b246a000 + 0xbb6 Raven.Server.RavenServerStartup+<RequestHandler>d__11.MoveNext()), calling (MethodDesc 00007ff7b246a000 + 0xdbf Raven.Server.RavenServerStartup+<RequestHandler>d__11.MoveNext())
000000BDCC4BF4C0 00007ff7b3dc0ec6 (MethodDesc 00007ff7b166d488 + 0x76 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000BDCC4BF530 00007ff7b484149f (MethodDesc 00007ff7b40e5e60 + 0xcf System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib],[Sparrow.Utils.ZstdStream+<ReadAsync>d__35, Sparrow]].MoveNext(System.Threading.Thread)), calling (MethodDesc 00007ff7b166d488 + 0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000BDCC4BF570 00007ff810b64fb0 coreclr!JIT_GetSharedGCThreadStaticBase + 0x70 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1830], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000BDCC4BF5B0 00007ff7b4354197 (MethodDesc 00007ff7b1d7dc60 + 0xa7 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
000000BDCC4BF5D0 00007ff7b4365a18 (MethodDesc 00007ff7b1962198 + 0x68 Raven.Server.RavenServerStartup.RequestHandler(Microsoft.AspNetCore.Http.HttpContext)), calling (MethodDesc 00007ff7b2ee1398 + 0 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.RavenServerStartup+<RequestHandler>d__11, Raven.Server]](<RequestHandler>d__11 ByRef))
000000BDCC4BF620 00007ff7b4348677 (MethodDesc 00007ff7b16350b0 + 0x77 System.Threading.Tasks.Task.RunContinuations(System.Object)), calling (MethodDesc 00007ff7b1d7dc60 + 0 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
000000BDCC4BF640 00007ff810b64fb0 coreclr!JIT_GetSharedGCThreadStaticBase + 0x70 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1830], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000BDCC4BF6C0 00007ff7b43e6035 (MethodDesc 00007ff7b1892170 + 0x55 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].TrySetResult(Int32)), calling (MethodDesc 00007ff7b1635098 + 0 System.Threading.Tasks.Task.FinishContinuations())
000000BDCC4BF6D0 00007ff7b1ce6e1a (MethodDesc 00007ff7b1d7b508 + 0x3a System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].AwaitUnsafeOnCompleted[[System.Runtime.CompilerServices.TaskAwaiter, System.Private.CoreLib]](System.Runtime.CompilerServices.TaskAwaiter ByRef, System.Runtime.CompilerServices.IAsyncStateMachineBox)), calling (MethodDesc 00007ff7b1634e10 + 0 System.Threading.Tasks.Task.UnsafeSetContinuationForAwait(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
000000BDCC4BF710 00007ff7b48412e3 (MethodDesc 00007ff7b40e5b50 + 0x83 System.Threading.Tasks.ValueTask`1+ValueTaskSourceAsTask+<>c[[System.Int32, System.Private.CoreLib]].<.cctor>b__4_0(System.Object)), calling (MethodDesc 00007ff7b1892170 + 0 System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib]].TrySetResult(Int32))
000000BDCC4BF730 00007ff810b4e9b3 coreclr!GCHandleManager::DestroyHandleOfUnknownType + 0x83 [D:\a\_work\1\s\src\coreclr\gc\gchandletable.cpp:129], calling coreclr!TableFreeSingleHandleToCache [D:\a\_work\1\s\src\coreclr\gc\handletablecache.cpp:768]
000000BDCC4BF770 00007ff7b4373ef7 (MethodDesc 00007ff7b2d60880 + 0x37 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean))
000000BDCC4BF7A0 00007ff7b434f0fb (MethodDesc 00007ff7b2d56730 + 0x8b System.Net.Sockets.SocketAsyncEventArgs.<CompleteCore>g__CleanupIOCPResult|171_0()), calling 00007ff810c290e0 (stub for System.Runtime.InteropServices.GCHandle.InternalFree(IntPtr))
000000BDCC4BF7E0 00007ff7b4373dbc (MethodDesc 00007ff7b2d60760 + 0x8c System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs)), calling (MethodDesc 00007ff7b2d60880 + 0 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean))
000000BDCC4BF7F0 00007ff7b3dc6673 (MethodDesc 00007ff7b2d562c8 + 0xd3 System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSyncSuccess(Int32, System.Net.Sockets.SocketFlags)), calling (MethodDesc 00007ff7b2d56730 + 0 System.Net.Sockets.SocketAsyncEventArgs.<CompleteCore>g__CleanupIOCPResult|171_0())
000000BDCC4BF800 00007ff7b3dc494b (MethodDesc 00007ff7b1ca0808 + 0x1b System.Threading.ThreadPoolBoundHandle.GetOverlappedWrapper(System.Threading.NativeOverlapped*)), calling 00007ff810c33ac0 (stub for System.Threading.OverlappedData.GetOverlappedFromNative(System.Threading.NativeOverlapped*))
000000BDCC4BF850 00007ff7b434ee90 (MethodDesc 00007ff7b2d63af8 + 0x130 System.Net.Sockets.SocketAsyncEventArgs+<>c.<.cctor>b__176_0(UInt32, UInt32, System.Threading.NativeOverlapped*))
000000BDCC4BF870 00007ff7b434ec4b (MethodDesc 00007ff7b1d7a070 + 0x1b System.Threading.ThreadPoolBoundHandleOverlapped.CompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)), calling 00007ff810c33ac0 (stub for System.Threading.OverlappedData.GetOverlappedFromNative(System.Threading.NativeOverlapped*))
000000BDCC4BF8B0 00007ff7b3e2de63 (MethodDesc 00007ff7b1ca3508 + 0x203 System.Threading.ThreadPoolTypedWorkItemQueue`2[[System.Threading.PortableThreadPool+IOCompletionPoller+Event, System.Private.CoreLib],[System.Threading.PortableThreadPool+IOCompletionPoller+Callback, System.Private.CoreLib]].System.Threading.IThreadPoolWorkItem.Execute())
000000BDCC4BF9A0 00007ff7b3dc10fa (MethodDesc 00007ff7b1ca0b58 + 0x13a System.Threading.ThreadPoolWorkQueue.Dispatch())
000000BDCC4BF9B8 00007ff7b3dc163e (MethodDesc 00007ff7b2cef7a0 + 0xde System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean)), calling coreclr!ThreadNative_UninterruptibleSleep0 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp:479]
000000BDCC4BFA20 00007ff7b4530cf4 (MethodDesc 00007ff7b2cef608 + 0x1d4 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()), calling (MethodDesc 00007ff7b1ca0b58 + 0 System.Threading.ThreadPoolWorkQueue.Dispatch())
000000BDCC4BFAA0 00007ff7b4530d42 (MethodDesc 00007ff7b2cef608 + 0x222 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()), calling coreclr!ThreadNative_UninterruptibleSleep0 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp:479]
000000BDCC4BFAB0 00007ff84154df75 combase!CoUninitialize + 0x135 [onecore\com\combase\class\compobj.cxx:3984], calling combase!__security_check_cookie [VCCRT\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000BDCC4BFB20 00007ff810c5fee3 coreclr!CallDescrWorkerInternal + 0x83
000000BDCC4BFB60 00007ff810b0eed0 coreclr!DispatchCallSimple + 0x60 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:221], calling coreclr!CallDescrWorkerWithHandler [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:54]
000000BDCC4BFB70 00007ff8415cdd56 combase!RoInitialize + 0x26 [onecore\com\combase\winrtbase\winrtbase.cpp:351], calling combase!RoInitializeForApartmentType [onecore\com\combase\winrtbase\winrtbase.cpp:293]
000000BDCC4BFBF0 00007ff810c508f3 coreclr!ThreadNative::KickOffThread_Worker + 0x63 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp:158], calling coreclr!DispatchCallSimple [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:171]
000000BDCC4BFC40 00007ff810bb6961 coreclr!ETW::ThreadLog::FireThreadCreated + 0x39 [D:\a\_work\1\s\src\coreclr\vm\eventtrace.cpp:2599], calling coreclr!FireEtwThreadCreated [D:\a\_work\1\s\artifacts\obj\coreclr\windows.x64.Release\inc\clretwallmain.h:1677]
000000BDCC4BFC50 00007ff810be9ec1 coreclr!ManagedThreadBase_DispatchMiddle + 0x85 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:7342], calling ntdll!LdrpDispatchUserCallTarget
000000BDCC4BFC60 00007ff810bb674c coreclr!ThreadStore::TransferStartedThread + 0x70 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:5468], calling coreclr!ThreadSuspend::UnlockThreadStore [D:\a\_work\1\s\src\coreclr\vm\threadsuspend.cpp:1914]
000000BDCC4BFC90 00007ff810bb6b40 coreclr!Thread::HasStarted + 0xf4 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:1884], calling coreclr!CLRException::HandlerState::CleanupTry [D:\a\_work\1\s\src\coreclr\vm\clrex.cpp:791]
000000BDCC4BFD30 00007ff810be9dd7 coreclr!ManagedThreadBase_DispatchOuter + 0xab [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:7501], calling coreclr!ManagedThreadBase_DispatchMiddle [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:7302]
000000BDCC4BFDD0 00007ff810be9cc9 coreclr!ThreadNative::KickOffThread + 0x79 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp:230], calling coreclr!ManagedThreadBase_DispatchOuter [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:7453]
000000BDCC4BFE30 00007ff841b77614 kernel32!BaseThreadInitThunk + 0x14, calling ntdll!LdrpDispatchUserCallTarget
000000BDCC4BFE60 00007ff8429a26b1 ntdll!RtlUserThreadStart + 0x21, calling ntdll!LdrpDispatchUserCallTarget
0:047> k
 # Child-SP          RetAddr               Call Site
00 (Inline Function) --------`--------     coreclr!MethodTable::GetFlag [D:\a\_work\1\s\src\coreclr\vm\methodtable.h @ 3522] 
01 (Inline Function) --------`--------     coreclr!MethodTable::HasComponentSize [D:\a\_work\1\s\src\coreclr\vm\methodtable.h @ 1522] 
02 (Inline Function) --------`--------     coreclr!MethodTable::IsStringOrArray [D:\a\_work\1\s\src\coreclr\vm\methodtable.h @ 1510] 
03 (Inline Function) --------`--------     coreclr!MethodTable::TestFlagWithMask [D:\a\_work\1\s\src\coreclr\vm\methodtable.h @ 3507] 
04 (Inline Function) --------`--------     coreclr!MethodTable::HasInstantiation [D:\a\_work\1\s\src\coreclr\vm\methodtable.inl @ 1021] 
05 (Inline Function) --------`--------     coreclr!Nullable::IsNullableForType+0x9 [D:\a\_work\1\s\src\coreclr\vm\object.inl @ 213] 
06 000000bd`cc4be420 00007ff8`10b0db3a     coreclr!ObjIsInstanceOfCore+0x8a [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 2097] 
07 000000bd`cc4be510 00007ff7`b3d7f106     coreclr!IsInstanceOfAny_NoCacheLookup+0x8a [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 2225] 
08 000000bd`cc4be670 00007ff7`b30b74f0     System_Private_CoreLib!System.Runtime.CompilerServices.CastHelpers.StelemRef_Helper_NoCacheLookup+0x16 [/_/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/CastHelpers.cs @ 623] 
09 000000bd`cc4be678 000002b2`8d36e0e8     0x00007ff7`b30b74f0
0a 000000bd`cc4be680 000002b3`27d24e30     0x000002b2`8d36e0e8
0b 000000bd`cc4be688 00000000`00000000     0x000002b3`27d24e30

image

Please let me know if additional information is required.

Reproduction Steps

First time it happened. Haven't tried to reproduce it yet. But we run in a cluster of 5 nodes and this happened only on one of them.

Expected behavior

Not to crash

Actual behavior

Process crashed

Regression?

Hard to tell, we constantly update to use the latest stable .net, but we do not run very often those stress tests and this is the first time it happened.

Known Workarounds

No response

Configuration

OS version: Windows 10 Version 19045 x64 CLR version: 7.0.923.32018

Other information

Reference to the code: https://github.com/ravendb/ravendb/blob/44a806981b4a4398b3b42bcf9604ac83d4df9d1d/src/Sparrow/Json/CachedProperties.cs#L324

mangod9 commented 1 year ago

we will look at the dump, but usually for such issues multiple dumps would be required to determine the root cause.

jandupej commented 1 year ago

Can this be related to https://[dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/398094/logs/541](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/398094/logs/541) which I got on Build Mono Product Build windows x64 release failed in https://github.com/dotnet/runtime/pull/91393?