microsoft / Detours

Detours is a software package for monitoring and instrumenting API calls on Windows. It is distributed in source code form.
MIT License
5.02k stars 981 forks source link

DetourFindFunction doesn't work where a direct call to SymFromName works. Why? #149

Closed MaxSavenkov closed 3 years ago

MaxSavenkov commented 3 years ago

I'm trying to detour a non-exported member method in a DLL using PDB (it might be important that the DLL's name differs from PDB name, and DLL has correct PDB name encoded inside).

When I try to use DetourFindFunction, a call to SymFromName inside it will return FALSE. However, if I use dbghelp API directly, I can get it to work. The difference seems to be in that I call SymLoadModule64 with BaseOfDll param set to 0 (so that it is taken from PDB), whether Detour always uses the result from LoadLibrary. If I change my call to SymLoadModule64 to use the result returned by a previous call to LoadLibrary, my call to SymFromName also fails.

I don't think this is a bug in Detour, but I can't understand what's happening and what am I doing wrong. I've looked at findfunc example, but it seems to me that I'm not doing anything too different. Basically, my current test project for Detour looks like this, I'm only trying to find the function:

#include "detours.h"
int main()
{
    // returns 0
    auto v4 = DetourFindFunction("c:\\x\\PathToDll.dll", "HiddenFunctionMangledName");
    return 0;
}

And my test project for dbghelp looks like this:

int main()
{
    HANDLE hProcess = GetCurrentProcess();
    DWORD64 BaseOfDll;
    BOOL status;

    status = SymInitialize(hProcess, NULL, FALSE);
    if (status == FALSE)
    {
        return -1;
    }    

    BaseOfDll = SymLoadModuleEx(hProcess,
        NULL,
        "c:\\x\\PathToDll.dll",
        NULL,
        0, // That's BaseOfDll parameter
        0,
        NULL,
        0);

    struct CFullSymbol : SYMBOL_INFO {
        CHAR szRestOfName[512];
    } symbol;
    ZeroMemory(&symbol, sizeof(symbol));
    symbol.SizeOfStruct = sizeof(SYMBOL_INFO);
    symbol.MaxNameLen = sizeof(symbol.szRestOfName) / sizeof(symbol.szRestOfName[0]);

    // returns 1, and symbol has correct symbol information
    auto v1 = SymFromName(hProcess, "DllName!HiddenFunctionMangledName", &symbol);

    return 0;
}

I even delved a bit into dbghelp disassembly, but all I got is that GetSymFromName calls FindModule, and it returns different results in these two cases, and later modGetSymFromName calls diaGetSymbols, which calls diaGetGlobals, passing the result from FindModule, and diaGetGlobals fails if the module was loaded with BaseOfDll != 0, but succeeds if it wasn't. It feels like PDB is not being loaded in the former case, and DIA is left with symbols from DLL itself.

I think I'm doing something wrong on my side, using Detour wrong, or something, but I can't quite figure out what. Can you help me?

MaxSavenkov commented 3 years ago

OK, I figured it out. For some reason, SymFromFile uses different search paths for PDBs depending of presence of BaseOfDll parameter: if it is not specified, is searches the same folder where DLL resides (along with system-defined folders), but if it is, it ignores DLL folder, and instead searches for PDB in current working directory.