Crash on Windows Server 2019 using Mimalloc v2.0.6

[vauki]

Hello! We recently started testing mimalloc v2.0.6 in our project and ran into an issue on Windows Server 2019. However, we don't have any issues on Win10/11. On startup we have a crash:

Faulting application name: launcher.exe, version: 0.0.0.0, time stamp: 0x632c2d38 Faulting module name: ucrtbase.dll, version: 10.0.17763.719, time stamp: 0x48ac8393 Exception code: 0xc0000409 Fault offset: 0x000000000006e77e Faulting process id: 0xebc Faulting application start time: 0x01d8ce7793646e97 Faulting application path: D:\bin\launcher.exe Faulting module path: C:\windows\System32\ucrtbase.dll Report Id: d62340f4-1b35-4f85-a29b-34ed8fccdff8 Faulting package full name: Faulting package-relative application ID:

Here is a callstack:

ntdll.dll!RtlReportFatalFailure() Unknown ntdll.dll!RtlReportCriticalFailure() Unknown ntdll.dll!RtlpHeapHandleError() Unknown ntdll.dll!RtlpHpHeapHandleError() Unknown ntdll.dll!RtlpLogHeapFailure() Unknown ntdll.dll!RtlFreeHeap() Unknown ucrtbase.dll!_free_base() Unknown [Inline Frame] rttr_core.dll!std::_Deallocate(void _Ptr, unsigned __int64 _Bytes) Line 255 C++ [Inline Frame] rttr_core.dll!std::allocator::deallocate(unsigned int const) Line 830 C++ [Inline Frame] rttr_core.dll!std::vector<unsigned int,std::allocator>::_Tidy() Line 1893 C++ [Inline Frame] rttr_core.dll!std::vector<unsigned int,std::allocator>::{dtor}() Line 791 C++ [Inline Frame] rttr_core.dll!std::_Vb_val<std::allocator>::{dtor}() Line 2658 C++ [Inline Frame] rttr_core.dll!std::vector<bool,std::allocator>::{dtor}() Line 2837 C++ rttr_core.dll!rttr::detail::constructor_wrapper_base::create_signature_string() Line 95 C++ rttr_core.dll!rttr::detail::constructor_wrapper_base::init() Line 58 C++ [Inline Frame] core.dll!rttr::detail::constructor_wrapper<engine::math::vec2,rttr::detail::class_ctor,0,rttr::detail::as_object,0,rttr::detail::default_args<>,rttr::detail::parameter_infos<rttr::detail::parameter_info_wrapper<float,0,rttr::detail::no_param_name,void>,rttr::detail::parameter_info_wrapper<float,1,rttr::detail::no_param_name,void>>,rttr::type_list<>,float,float>::{ctor}(std::array<rttr::detail::metadata,0>) Line 88 C++ [Inline Frame] core.dll!rttr::detail::make_unique(std::array<rttr::detail::metadata,0> &&) Line 276 C++ core.dll!rttr::registration::bind<rttr::detail::ctor,engine::math::vec2,rttr::detail::public_access,rttr::type_list<>,float,float>::create_constructor_wrapper<rttr::detail::as_object,0,rttr::detail::parameter_info_wrapper<float,0,rttr::detail::no_param_name,void>,rttr::detail::parameter_info_wrapper<float,1,rttr::detail::no_param_name,void>>(std::array<rttr::detail::metadata,0> metadata_list, rttr::detail::default_args<> def_args, rttr::detail::parameter_infos<rttr::detail::parameter_info_wrapper<float,0,rttr::detail::no_param_name,void>,rttr::detail::parameter_info_wrapper<float,1,rttr::detail::no_param_name,void>> param_infos) Line 133 C++ [Inline Frame] core.dll!rttr::registration::bind<rttr::detail::ctor,engine::math::vec2,rttr::detail::public_access,rttr::type_list<>,float,float>::operator()(const rttr::detail::as_object &) Line 202 C++ core.dll!engine::rttr_auto_register_reflectionfunction() Line 106 C++ ucrtbase.dll!_initterm() Unknown core.dll!dllmain_crt_process_attach(HINSTANCE__ const instance, void const reserved) Line 66 C++ core.dll!dllmain_dispatch(HINSTANCE__ const instance, const unsigned long reason, void const reserved) Line 276 C++ ntdll.dll!LdrpCallInitRoutine() Unknown ntdll.dll!LdrpInitializeNode() Unknown ntdll.dll!LdrpInitializeGraphRecurse() Unknown ntdll.dll!LdrpInitializeGraphRecurse() Unknown ntdll.dll!LdrpInitializeGraphRecurse() Unknown ntdll.dll!LdrpInitializeGraphRecurse() Unknown ntdll.dll!LdrpInitializeGraphRecurse() Unknown ntdll.dll!LdrpInitializeProcess() Unknown ntdll.dll!_LdrpInitialize() Unknown ntdll.dll!LdrpInitialize() Unknown ntdll.dll!LdrInitializeThunk() Unknown

I have already tried the fixes suggested in https://github.com/microsoft/mimalloc/issues/521 and in https://github.com/microsoft/mimalloc/issues/103. Such as link directly with mimalloc-redirect.lib and use /INCLUDE:mi_redirect_enable. As well as apply minject.exe after. These steps did not help.

I think the problem is with ucrtbase.dll. We are using version 10.0.17763.719.

PS. I didn't find the source code for malloc-redirect. Could you provide it?

[daanx]

Hi -- thanks for the report. It looks like mimalloc redirection actually failed as it would normally intercept _free_base but here it invoked RtlFreeHeap. Can you set the environment variable MIMALLOC_VERBOSE=3 and then run the program and send me the output? This will give very verbose information about the redirection.

Also, ensure mimalloc-redirect.dll is in the same directory as the program. Thanks!

[vauki]

I researched a problem a little in a Debug build and found that the crash occurs when the RtlGenRandom() function is called (inside os_random_buf). I switched to using BCryptGenRandom and it stopped crashing.

[panikc]

After updating mimalloc from v1.7.5 to v2.0.6 in our software we also get a crash on startup when running on Windows Server 2019. The call stack is consistent with @vauki's analysis in the previous post, also pointing to the usage of RtlGenRandom in os_random_buf:

ucrtbase.dll!abort()
VCRUNTIME140.dll!__vcrt_getptd() Line 166
VCRUNTIME140.dll!__current_exception() Line 115
VCRUNTIME140_1.dll!__vcrt_getptdbase() Line 185
VCRUNTIME140_1.dll!__CxxFrameHandler4(EHExceptionRecord * pExcept, unsigned __int64 RN, _CONTEXT * pContext, _xDISPATCHER_CONTEXT * pDC) Line 296
ntdll.dll!RtlpExecuteHandlerForException()
ntdll.dll!RtlDispatchException()
ntdll.dll!KiUserExceptionDispatch()
bcryptPrimitives.dll!ProcessPrng()
CRYPTBASE.DLL!SystemFunction036()
[Inline Frame] mimalloc.dll!os_random_buf(void *) Line 185
mimalloc.dll!_mi_random_init(mi_random_cxt_s * ctx) Line 306
mimalloc.dll!mi_heap_main_init() Line 170
mimalloc.dll!mi_process_init() Line 579
mimalloc.dll!mi_thread_init() Line 426
mimalloc.dll!_mi_malloc_generic(mi_heap_s * heap, unsigned __int64 size) Line 840
[Inline Frame] mimalloc.dll!_mi_heap_malloc_zero(mi_heap_s *) Line 152
[Inline Frame] mimalloc.dll!mi_heap_zalloc(mi_heap_s *) Line 159
mimalloc.dll!mi_heap_calloc(mi_heap_s * heap, unsigned __int64 count, unsigned __int64 size) Line 608
ucrtbase.dll!__acrt_initialize_ptd()
ucrtbase.dll!DllMainProcessAttach()
ucrtbase.dll!DllMainDispatch()
ntdll.dll!LdrpCallInitRoutine()
ntdll.dll!LdrpInitializeNode()
ntdll.dll!LdrpInitializeGraphRecurse()
ntdll.dll!LdrpInitializeGraphRecurse()
ntdll.dll!LdrpInitializeGraphRecurse()
ntdll.dll!LdrpInitializeGraphRecurse()
ntdll.dll!LdrpInitializeGraphRecurse()
ntdll.dll!LdrpInitializeProcess()
ntdll.dll!_LdrpInitialize()
ntdll.dll!LdrpInitialize()
ntdll.dll!LdrInitializeThunk()

[daanx]

Thanks for the logs! Hmm, what a strange and annoying problem -- but glad we figured out what causes it. I would prefer to just always switch to use BCryptGenRandom as that is the official API but the comment in the source code seems to imply that there is some issue with it when debugging from Visual Studio. I will investigate this first again.
(for now, recompiling mimalloc to use BCryptGenRandom is a good work around)

[vauki]

..but the comment in the source code seems to imply that there is some issue with it when debugging from Visual Studio. I will investigate this first again.

Yes, I saw this comment as well. I am using MSVC 2022 and after switching to BCryptGenRandom I did not encounter any problems.

[Chronial]

Did you consider using rand_s? It's also what the MS STL uses: https://github.com/microsoft/STL/blob/bafb10ea0dd21efb27b3f2346f898fef52f79235/stl/src/xrngdev.cpp#L12