x64 unhooking problem in MT application

Hello guys,

I encountered a problem with unhooking in the next scenario. There is a simple native x64 application:

#include <iostream>
#include <Windows.h>
#include <process.h>
#include "NktHookLib.h"

typedef int (*TheFunctionFunc)(void*);

CNktHookLib::HOOK_INFO g_hook;

int TheFunctionHook(void* p1) {
    std::cout << "Hook" << std::endl;

    if (g_hook.lpCallOriginal) {
        return ((TheFunctionFunc)(g_hook.lpCallOriginal))(p1);
    }

    return 0;
}

TheFunctionFunc TheFunction;

void init() {
    HMODULE hModule = LoadLibrary(L"module.dll");

    if (!hModule)
        return;

    TheFunction = (TheFunctionFunc)GetProcAddress(hModule, "TheFunction");
}

CNktHookLib g_lib;

void install_hooks() {
    g_lib.SetEnableDebugOutput(TRUE);
    g_hook.lpProcToHook = TheFunction;
    g_hook.lpNewProcAddr = TheFunctionHook;
    g_lib.Hook(&g_hook, 1);
}

void uninstall_hooks() {
    g_lib.Unhook(&g_hook, 1);
}

unsigned __stdcall run(void*) {
    while (true) {
        TheFunction(nullptr);
    }

    return 0;
}

int main() {
    init();
    TheFunction(nullptr);
    HANDLE hThread = (HANDLE)_beginthreadex(nullptr, 2048, run, nullptr, 0, 0);

    while (true) {
        install_hooks();
        //WaitForSingleObject(hThread, 61000);
        uninstall_hooks();
        //WaitForSingleObject(hThread, 1000);
    }

    CloseHandle(hThread);
}

This code lead to crash of the application during uninstallation of the hook in the _beginthreadex thread. Analyzing disassembly code and NktHookLib.cpp revealed next.

            *p++ = 0x48;  *p++ = 0xF7;  *p++ = 0x02;                             //test  QWORD PTR [rdx], 00000101h
            *((ULONG NKT_UNALIGNED*)p) = 0x00000101;
            p += sizeof(ULONG);
            //----
            *p++ = 0x75;                                                         //jne   CALL_ORIGINAL
            *p++ = ((lpHookEntry->dwFlags & NKTHOOKLIB_DisallowReentrancy) != 0) ? 0x7A : 0x06;
            //check for reentranct

Which as I understood lead to this code:

000007FEBA6A0022 75 06                jne         000007FEBA6A002A  
000007FEBA6A0024 5A                   pop         rdx  
000007FEBA6A0025 FF 25 00 00 00 00    jmp         qword ptr [7FEBA6A002Bh]  
000007FEBA6A002B 4B 10 FC             adc         r12b,dil  
000007FEBA6A002E ??                   db          3fh  
000007FEBA6A002F 01 00                add         dword ptr [rax],eax  
000007FEBA6A0031 00 00                add         byte ptr [rax],al  
000007FEBA6A0033 5A                   pop         rdx  
000007FEBA6A0034 4C 89 4C 24 20       mov         qword ptr [rsp+20h],r9  
000007FEBA6A0039 FF 25 00 00 00 00    jmp         qword ptr [7FEBA6A003Fh]

but jump 000007FEBA6A002A is not correct in case of unloading. The correct address I suppose is 000007FEBA6A0033 thus offset 0x06 should be replaced with 0x0F. Doing this fixed the problem on my side. So, if you have a chance to review this place that would be great.

Thanks, Mikalaj

nektra / Deviare-InProc

x64 unhooking problem in MT application #3