HeapAlloc hooking, deadlock on Windows 10, works on Windows 7

This post is also on StackOverflow. I have tested with the release version of minihook and the git master version. It is maybe a minihook bug. Any ideas?

###### I have a most peculiar bug... I'm hooking HeapAlloc to log all calls and get the name the DLLs calling the API. The code works on Windows 7, but doesn't work on Windows 10. I use miniHook for hooking. Everything compiled with Visual Studio 2019, v142.

// BasicTest.cpp : This file contains the 'main' function. Program execution begins and ends there.
//
#include <windows.h>
#include <iostream>
#include <intrin.h>
#include <psapi.h>
#include "minihook.h"

#pragma intrinsic(_ReturnAddress)
#pragma comment(lib, "libMinHook.x64.lib") //v142

LPVOID(WINAPI* OldHeapAlloc)(HANDLE, DWORD, SIZE_T);

template <typename T>
inline MH_STATUS MH_CreateHookApiEx(
    LPCWSTR pszModule, LPCSTR pszProcName, LPVOID pDetour, T** ppOriginal)
{
    return MH_CreateHookApi(
        pszModule, pszProcName, pDetour, reinterpret_cast<LPVOID*>(ppOriginal));
}

BOOL intercept = FALSE;
int iMbox = 0;
int iTotal = 0;

LPVOID HookedHeapAlloc(HANDLE hHeap, DWORD dwFlags, SIZE_T dwBytes) {
    iTotal++;
    LPVOID ptr = OldHeapAlloc(hHeap, dwFlags, dwBytes);
    if (intercept) {
        return ptr;
    }

    intercept = TRUE;
    iMbox++;

    HMODULE hModule;
    char lpBaseName[32];

    if (GetModuleHandleExA(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS, (LPCSTR)_ReturnAddress(), &hModule) != 0) {
        if (GetModuleBaseNameA(GetCurrentProcess(), hModule, lpBaseName, sizeof(lpBaseName)) != 0) {
            printf("Reserved %d at %08x from %s\n", dwBytes, ptr, lpBaseName);
        }
    }
    intercept = FALSE;

    return ptr;
}

int main()
{
    if (MH_Initialize() != MH_OK)
    {
        return 1;
    }

    if (MH_CreateHookApiEx(
        L"ntdll.dll", "RtlAllocateHeap", &HookedHeapAlloc, &OldHeapAlloc) != MH_OK)
    {
        return 1;
    }

    if (MH_EnableHook(MH_ALL_HOOKS) != MH_OK)
    {
        return 1;
    }

    MessageBoxA(NULL, "This is a test", "Test", MB_OK);

    MH_DisableHook(MH_ALL_HOOKS);
    printf("RtlAllocateHeap calls for MessageBoxA = %d\n", iMbox);
    printf("RtlAllocateHeap total calls  = %d\n", iTotal);

    return 0;
}

intercept is to prevent recurrence inside HookedHeapAlloc. Because GetModule* like functions call HeapAlloc. It is important to note that the code works if you call HeapAlloc yourself, even with recurrence (HeapAlloc -> HeapAlloc -> HeapAlloc). You can call HeapAlloc 5000 times, and put a depth of 5 recursion inside HookedHeapAlloc (intercept prevents any crash).

But when using MessageBoxA, the program hangs on Windows 10 (tested in 20H2 and 21H1).

The output on Windows 7 is something like that

[.......]
Reserved 24 at 004528f0 from KERNELBASE.dll
Reserved 40 at 0046afb0 from KERNELBASE.dll
Reserved 520 at 02aae4f0 from uxtheme.dll
Reserved 512 at 0046dd90 from IMM32.DLL
Reserved 48 at 00468960 from ntdll.dll
Reserved 48 at 004689a0 from ntdll.dll
Reserved 512 at 004612a0 from USER32.dll
Reserved 48 at 004689e0 from ntdll.dll
Reserved 48 at 00468a20 from ntdll.dll
Reserved 48 at 00468a60 from ntdll.dll
RtlAllocateHeap calls for MessageBoxA = 828
RtlAllocateHeap total calls  = 1657

On Windows 10, the program hangs after a couple of outputs. When inside a debugger, it hangs at ZwWaitForAlertByThreadId.

Thanks for the help :)

TsudaKageyu / minhook

HeapAlloc hooking, deadlock on Windows 10, works on Windows 7 #99