VLD infinite method call loop

[WildCard65]

VLD runs into an infinite loop when it calls TlsGetValue during it's HeapAlloc hook

[KindDragon]

All test passing. Can you provide more information? Minimum reproducing example?

[WildCard65]

Basically, I want to test a DLL I'm making for 2 projects (AMXMDOX and SourceMod), my DLL is loaded at run time via LoadLibraryA, my DLL also (if I understand VLD api correctly) enables itself on DLLMain ProccessAttach for (again hopefully) main thread only (as that's only thread all code is executed on), but after DllMain call, srcds (Testing with SM atm) crashes from stack overflow, the dll causing crash is mostly the nt dll and some occasions it's vld_x86.dll (srcds is 32bit)

[WildCard65]

Here's a copy + paste of call stack at time of stack overflow:

/* Truncated A LOT of repeats of the next 4 lines */
Line 176 + 0x1a bytes   C++
    KernelBase.dll!_TlsSetValue@8()  + 0x72 bytes   
    vld_x86.dll!VisualLeakDetector::getTls()  Line 1305 C++
    vld_x86.dll!VisualLeakDetector::_HeapAlloc(void * heap, unsigned long flags, unsigned long size)  Line 176 + 0x1a bytes C++
    KernelBase.dll!_TlsSetValue@8()  + 0x72 bytes   
    vld_x86.dll!VisualLeakDetector::getTls()  Line 1305 C++
    vld_x86.dll!VisualLeakDetector::_HeapAlloc(void * heap, unsigned long flags, unsigned long size)  Line 176 + 0x1a bytes C++
    msvcrt.dll!__calloc_impl()  + 0x136 bytes   
    msvcrt.dll!__calloc_crt()  + 0x16 bytes 
    msvcrt.dll!__CRTDLL_INIT@12()  + 0xfc bytes 
>   vld_x86.dll!LdrpCallInitRoutine(void * BaseAddress, unsigned long Reason, void * Context, unsigned char (void *, unsigned long, _CONTEXT *)* EntryPoint)  Line 98 + 0xc bytes   C++
    ntdll.dll!_LdrxCallInitRoutine@16()  + 0x16 bytes   
    ntdll.dll!LdrpCallInitRoutine()  + 0x43 bytes   
    ntdll.dll!_LdrpInitializeThread@4()  + 0x2a7 bytes  
    ntdll.dll!__LdrpInitialize@8()  + 0x30f bytes   
    ntdll.dll!_LdrpInitialize@8()  + 0x25 bytes 
    ntdll.dll!_LdrInitializeThunk@8()  + 0x10 bytes 

[WildCard65]

Could me running Windows 10 be the reason this happened or no?

[KindDragon]

I have Win10 too. Can you try build VLD and run all test?

@ioannis-e Maybe you remember something about that?

[WildCard65]

@KindDragon I ran Win32 Debug-v140 tests and results: All passed
This leaves me to believe the issue relies on how VLD is being loaded on my end. VLD is being late loaded along side my project's DLL (Late loaded being loaded via LoadLibraryA after EXE successfully started)

[WildCard65]

OFF-TOPIC: Accidently clicked "Close" by accident :/

[WildCard65]

Adding onto this, I believe this may be the best case min-repro of this issue: 1) Create an EXE that loads a DLL at runtime (LoadLibrary call) 2) Have the DLL that will be loaded require VLD. 3) EXE should crash.

[KindDragon]

We have tested for this, but it pass on my system

[WildCard65]

Ok, is it possible be an issue 32 bit wise? My project was loaded by SourceMod which is loaded by Metamod:Source which is loaded by srcds. Also should note SM does vtable hooks and method detours on some methods at runtime.

[WildCard65]

Ok, I narrowed the actual cause: TlsSetValue -> RtlAllocateHeap call -> VisualLeakDetector::_HeapAlloc -> VisualLeakDetector::enabled -> VisualLeakDetector::getTls -> TlsGetValue returning null -> loop start

[WildCard65]

In more english terms @KindDragon VLD is stuck trying to check if it's enabled by trying to set Thread Local Storage after hooking RtlAllocateHeap

[KindDragon]

Also should note SM does vtable hooks and method detours on some methods at runtime.

We also hook some system functions, perhaps because of this.

[WildCard65]

SM's VTable hooking/detouring generally applies to srcd's engine.dll and the game's server.dll Reference: https://github.com/alliedmodders/sourcemod/tree/master/gamedata

Edit: Some hooking is gamedata less, but generally as far as I know, SM doesn't hook any alloc functions.

[ioannis-e]

@WildCard65 can you confirm that m_tlsIndex = TlsAlloc(); returns a valid index, and that m_tlsIndex != TLS_OUT_OF_INDEXES

Also can you advice how many threads are created at the point of the infinite loop ?

[WildCard65]

@ioannis-e At start of VLD, the check for TLS_OUT_OF_INDEXES passes in the constructor, at time of infinite loop, 26 threads are created.

Edit: 4 out of the 26 threads belong to SourceMod, it should also be noted that SRCDs is designed as single threaded, so by inheritence, MM:S and SM are single threaded. And SM is responsible for loading my project.

[WildCard65]

@KindDragon Could it be possible because it's being loaded in an environment more than likely compiled with the static version of the CRT library while VLD isn't?

[ImmortalBLG]

Any news about solution of stack overflow?

[RARain]

Same problem.

[Junch]

Please view the link https://xinhuang.github.io/posts/2015-10-23-visual-leak-detector-stack-overflow-and-thread-local-storage.html

The author seemed to have a fix. But clearly the fix is not applied to the current release.