search

microsoft / Detours

Detours is a software package for monitoring and instrumenting API calls on Windows. It is distributed in source code form.
MIT License
5k stars 978 forks source link

Returns ERROR_INVALID_BLOCK on large functions with __chkstk on it. #254

Open 0Camus0 opened 1 year ago

0Camus0 commented 1 year ago

Latest version as for today(Version 4.0.1).

Trying to hook to a large function, looking at the disassembly:

     9: void function()
    10: {
00007FF6E296E010 B8 C8 6C 00 00      mov         eax,6CC8h  
00007FF6E296E015 E8 A3 3B FE FF       call        __chkstk (07FF6E2951BBDh)  
00007FF6E296E01A 48 2B E0                sub         rsp,rax

I have the same project built at home and the __chkstk is not there, deours hook without issues. At another PC with the same compiler, it builds adding that instruction and detour fails to hook.

I assume the block is too small thanks to that call to __chkstk. I think it should be supported, but still, please let me know if this is an issue or if it's expected.

Building with VS2022 target x64, compiler options: /MD /Zi /O2 /Ob1 /DNDEBUG

Thanks.