Support other calling conventions?

[callym]

I'm interested in using this for trying to hook into an x86 Windows game.

Looking through the assembly, it looks like it uses a combination of cdecl, thiscall, and stdcall calling conventions, but it looks like ilhook only supports cdecl?

I'm happy to attempt to work on this as a PR if you had some pointers to where to start.

[regomne]

Ilhook supports cdecl in the hook routine (such as JmpBackRoutine), but the type of hook routine has nothing to do with the target function to be hooked. It can hook target functions of all calling convention types.

For example, a thiscall function:

class foo {
public:
    virtual void set_bar(int bar) { bar_ = bar; }
    int bar_;
}

If you want to get the value of bar_ when foo::set_bar is being called, you can hook set_bar function and get it from this pointer in ecx:

unsafe extern "C" fn on_set_bar(
    reg: *mut Registers,
    _: usize
) {
    let this_ptr = (*reg).ecx; // thiscall
    let bar_ptr = (this_ptr + 4) as *const u32; // bar should be in offset 4
    println!("bar: {}", *bar_ptr);
}

Hooker.hook(SET_BAR_ADDRESS, HookType::JmpBack(on_set_bar), CallbackOptions::None, HookFlags::empty())?;

[callym]

This is super useful, thanks! I opened a PR - the crashes I was getting thinking it was calling-convention related were actually to do with encoding/decoding 64-bit instructions in the 32-bit module (I think?)