Function Hooking deadlocks

[vars1ty]

In Rust if you hook a function with all of the correct parameters, it works as long as you don't combine the hooked function with any custom code, which kinda defeats the purpose of hooking in the first place.

The deadlock seems rather random, as it works just fine on one function that I've got, whereas on another I can't do anything but println!() before it locks itself.

To reproduce, try hooking functions in a game like Assault Cube and add more logic to the hooked function, by for example reading a string from a parameter, assuming it's the correct type.

Edit: Note, this is on x86.

[rdbo]

What is the code of your hook function? You have to use C types instead of Rust times if you want to extract information from the parameters. Also, IIRC, some (not many TBH) Assault Cube functions have weird calling conventions, so you might have to do inline ASM.

[vars1ty]

Sorry for the late response. I'm not currently using Assault Cube, switched to a different game to test and it's experiencing deadlocks when calling additional logic inside of the target function, which is being used for the hook.

Here's the game I'm currently using: https://www.swisstransfer.com/d/a2eb7459-6ebd-46ef-b3b0-3c45ffdcb422 Function Address: 0x004935d0 Parameters: *const i8, c_int, c_int

I've been hooking this function using retour-rs for several months, and it's been working just fine. But I need to go external for several other games, and decided to try and hook the function above and it just locks upon receiving calls.

The additional code I had in it wasn't anything special, it would deadlock from simply just having a for-loop in it and a println!(), then calling the original function after everything was done.

[rdbo]

Oh, I just remembered. I actually have a working hook with Rust for AssaultCube on Linux (don't mind the messy code BTW, this was just a test): https://github.com/rdbo/rustycube/blob/master/src/lib.rs#L57-L64

This is a hook for gl_drawframe tested on Linux.

But I need to go external for several other games, and decided to try and hook the function above and it just locks upon receiving calls.

External hooks require you to inject your hook function into the process's memory space beforehand. I generally don't recommend doing it, because it's very easy to get it wrong. Also, internal allows you to work with the target process with much more ease.

[rdbo]

Maybe this is a Windows problem? I'll have to check it out.

[vars1ty]

External hooks require you to inject your hook function into the process's memory space beforehand. I generally don't recommend doing it, because it's very easy to get it wrong. Also, internal allows you to work with the target process with much more ease.

Ah, except this was using the internal method, with the DLL injected, so it should of worked it seems like without any issues. I'm on Arch Linux and doing everything by cross-compiling the DLL via MSVC, then injecting it via a custom injector.

[rdbo]

This seems to be a Windows problem. Looking into it.

[rdbo]

The hooking tests have passed on Windows now - the Windows error that was happening before was "unrelated" (MSVC generates a dummy function, so the hook was being placed in the wrong address). On Linux, hooks have been working from the start. I will close this for now.