Support new version of dbghelp.dll

[z16166]

The 6.11.1.404 version of dbghelp.dll can't deal with some pdb details produced by VS2019/VS2022. It randomly crashes when reporting leaks.

I just build VLD from souce code, with the version 10 dbghelp.dll and debug it. It seems that SymInitialize( ) of version 10 dbghelp.dll will load new dll, so it will call g_vld.RefreshModules() -> attachToLoadedModules() -> SymLoadModuleExW().

That means SymInitialize( ) will call SymLoadModuleExW( ), which breaks Microsoft API, because SymLoadModuleExW( ) requires that SymInitialize( ) succeeds first.

So we have to find a workaround for this, for example, postponne the call to symbol API SymInitialize/SymLoadModuleExW to the reporting phrase?

[z16166]

Well, there is an easy solution to prevent SymInitialize() from calling SymLoadModuleExW( ). Just add a global bool variable to indicate that we are calling SymInitialize(). Tested working with: VS2019 + Win11 insider dev channel + c:\windows\system32\dbghelp.dll.

Rember to remove "vld.dll.dependency.x64.manifest" and "dbghelp.dll" and "Microsoft.DTfW.DHL.manifest" from vld project settings to use c:\windows\system32\dbghelp.dll which is version 10.0.

dbghelp.h:

extern volatile bool init;

BOOL SymInitializeW(_In_ HANDLE hProcess, _In_opt_ PCWSTR UserSearchPath, _In_ BOOL fInvadeProcess) {
      init = true;

        CriticalSectionLocker<CriticalSection> cs(m_lock);
        const auto r = ::SymInitializeW(hProcess, UserSearchPath, fInvadeProcess);
        init = false;
        return r;
    }

vld.cpp

volatile bool init = false;

typedef BOOLEAN(NTAPI *PDLL_INIT_ROUTINE)(IN PVOID DllHandle, IN ULONG Reason, IN PCONTEXT Context OPTIONAL);
BOOLEAN WINAPI LdrpCallInitRoutine(IN PVOID BaseAddress, IN ULONG Reason, IN PVOID Context, IN PDLL_INIT_ROUTINE EntryPoint)
{
    LoaderLock ll;

    if (Reason == DLL_PROCESS_ATTACH) {
      if (!init)
        g_vld.RefreshModules();
    }

    return EntryPoint(BaseAddress, Reason, (PCONTEXT)Context);
}