Need tutorial

[AurevoirXavier]

Any tutorial? Can I hook d3d11 with this crate?

[darfink]

Yes it's possible to detour D3D11 methods and functions with this library. It might be worth mentioning though that VMT (Virtual Method Table) hooks are most commonly used for Direct3D APIs.

If inline hooking is to be used (which this library provides), you need to be aware that the D3D11 methods are defined as STDMETHODCALLTYPE and therefore uses the stdcall calling convention on x86 (but not on x64).

So an equivalent for ID3D11DeviceContext::Draw would be:

static_detours! {
    struct ID3D11DeviceContextDraw: extern "system" fn(*const (), u32, u32);
}

Where the first argument is the ID3D11DeviceContext instance (obviously you would want to define this type instead of using ()).

To initialise the detour you would need to obtain an address to the ID3D11DeviceContext::Draw method.

[AurevoirXavier]

Thanks. Could you give some examples, something like hook notepad's messagebox ? That would be a great help.

[darfink]

Here's an example repository from another user: https://github.com/Verideth/dll_hook-rs

[AurevoirXavier]

It seems that hook a function in a game which use source engine? I'm not sure where to inject it. It's a good code example, but I don't know what the dll actually do.

[darfink]

The linked example and the this repository's README both illustrates ways of using this library. I suspect you have issues understanding concepts that this library does not cover.

Let's put it this way, what have you tried and where do you get stuck?

• Is it how to define the static detour?
• How to initialize the detour?
• How to call the original function?
• How to retrieve the address of a function to be hooked?
• How to inject a DLL?

I do not have access to a Windows computer but hooking MessageBoxW should be something along these lines (I haven't compiled this code, it's merely a guide):

static_detours! {
  struct DetourMessageBoxW: unsafe extern "system" fn(HWND, LPCWSTR, LPCWSTR, UINT) -> c_int;
}

unsafe fn messageBoxDetour(handle: HWND, text: LPCWSTR, caption: LPCWSTR, uType: UINT) -> c_int {
  // Simply forward the call to the original
  DetourMessageBoxW.get().unwrap().call(handle, text, caption, uType)
}

#[no_mangle]
#[allow(non_snake_case, unused_variables)]
pub extern "system" fn DllMain(dll_module: HINSTANCE, call_reason: DWORD, reserved: LPVOID) -> BOOL {
    const DLL_PROCESS_ATTACH: winapi::DWORD = 1;
    if call_reason == DLL_PROCESS_ATTACH {
      let mut hook = unsafe { DetourMessageBoxW.initialize(MessageBoxW, messageBoxDetour).unwrap() };
      unsafe { hook.enable().unwrap() };

      // Omit the drop to persist the hook after the current scope.
      // A better alternative is to define a variable with a longer lifetime.
      mem::forget(hook);
    }

    return TRUE;
}

[AurevoirXavier]

I just don't know how to start it. And I want a demo to run it. I try to make a MyMessageBox but I don't know the right operation flow.

Thanks for your patient. But DetourMessageBoxW.initialize() need a closure, can't pass messageBoxDetour to it.

[darfink]

Ah yes, my bad. The Fn trait is not implemented for unsafe nor non-rust extern functions. The problem can be fixed by removing the unsafe modifier:

fn messageBoxDetour(handle: HWND, text: LPCWSTR, caption: LPCWSTR, uType: UINT) -> c_int {
  // Simply forward the call to the original
  unsafe { DetourMessageBoxW.get().unwrap().call(handle, text, caption, uType) }
}

Of course you can also pass a closure directly to initialize:

DetourMessageBoxW.initialize(MessageBoxW, |handle, text, caption, uType| {
  unsafe { DetourMessageBoxW.get().unwrap().call(handle, text, caption, uType) }
})

[AurevoirXavier]

#[macro_use]
extern crate detour;
extern crate winapi;

// --- std ---
use std::mem;
// --- external ---
use winapi::{
    ctypes::c_int,
    shared::{
        windef::HWND,
        minwindef::{BOOL, DWORD, HINSTANCE, LPVOID, UINT, TRUE},
    },
    um::{
        winnt::LPCWSTR,
        winuser::MessageBoxW,
    },
};

static_detours! {
  struct DetourMessageBoxW: unsafe extern "system" fn(HWND, LPCWSTR, LPCWSTR, UINT) -> c_int;
}

fn message_boxw_detour(h_wnd: HWND, _: LPCWSTR, lp_caption: LPCWSTR, u_type: UINT) -> c_int {
    // Simply forward the call to the original
    let text: Vec<u16> = "Ops, hook by detour-rs.".encode_utf16().collect();
    unsafe { DetourMessageBoxW.get().unwrap().call(h_wnd, text.as_ptr() as _, lp_caption, u_type) }
}

#[no_mangle]
#[allow(non_snake_case, unused_variables)]
pub extern "system" fn DllMain(dll_module: HINSTANCE, call_reason: DWORD, reserved: LPVOID) -> BOOL {
    const DLL_PROCESS_ATTACH: DWORD = 1;
    if call_reason == DLL_PROCESS_ATTACH {
        let mut hook = unsafe { DetourMessageBoxW.initialize(MessageBoxW, message_boxw_detour).unwrap() };
        unsafe { hook.enable().unwrap() };

        // Omit the drop to persist the hook after the current scope.
        // A better alternative is to define a variable with a longer lifetime.
        mem::forget(hook);
    }

    return TRUE;
}

I wrote a program which can pop up a MessageBoxW, and I inject the dll with extreme injector. But the message doesn't change.

[darfink]

It was a long time since I performed API hooking on Windows, but after some trial and error I've identified the cause. Each DLL gets a separate import address table which is mapped at runtime. This leads to an indirection, and due to this any imported function addresses differs from the executable's.

To solve this you can obtain an absolute address using GetModuleHandleW & GetProcAddress:

pub fn lookup(module: &str, symbol: &str) -> Option<usize> {
    let mut module: Vec<u16> = module.encode_utf16().collect();
    module.push(0);
    let symbol = CString::new(symbol).unwrap();
    unsafe {
        let handle = GetModuleHandleW(module.as_ptr());
        match GetProcAddress(handle, symbol.as_ptr()) as usize {
            0 => None,
            n => Some(n),
        }
    }
}

This can be utilized as follows:

type FnMessageBoxW = unsafe extern "system" fn(HWND, LPCWSTR, LPCWSTR, UINT) -> c_int;

#[no_mangle]
#[allow(non_snake_case, unused_variables)]
pub extern "system" fn DllMain(dll_module: HINSTANCE, call_reason: DWORD, reserved: LPVOID) -> BOOL {
    const DLL_PROCESS_ATTACH: DWORD = 1;
    if call_reason == DLL_PROCESS_ATTACH {
        // Retrieve an absolute address of `MessageBoxW`
        let address = lookup("user32.dll", "MessageBoxW").unwrap();
        let target: FnMessageBoxW = unsafe { mem::transmute(address) };

        let mut hook = unsafe { DetourMessageBoxW.initialize(target, message_boxw_detour).unwrap() };
        unsafe { hook.enable().unwrap() };

        // Omit the drop to persist the hook after the current scope.
        // A better alternative is to define a variable with a longer lifetime.
        mem::forget(hook);
    }

    return TRUE;
}

Note also that you are missing a null terminator when you are creating the text parameter in the message_boxw_detour function.

fn message_boxw_detour(h_wnd: HWND, _: LPCWSTR, lp_caption: LPCWSTR, u_type: UINT) -> c_int {
    // Simply forward the call to the original
    let mut text: Vec<u16> = "Ops, hook by detour-rs.".encode_utf16().collect();
    text.push(0); // <-- this is essential
    unsafe { DetourMessageBoxW.get().unwrap().call(h_wnd, text.as_ptr() as _, lp_caption, u_type) }
}

Here is a complete and verified code sample: gist.

[AurevoirXavier]

Thanks!