MessageBox hook

[4zv4l]

use retour::static_detour;
use windows::{s,core::{PCSTR}, Win32::UI::WindowsAndMessaging::*, Win32::Foundation::*};

static_detour! {
    static HookMessageBox: unsafe fn(HWND, PCSTR, PCSTR, MESSAGEBOX_STYLE) -> MESSAGEBOX_RESULT;
}

#[test]
fn hooking_get_message() {
    unsafe {
        MessageBoxA(None, s!("Ansi"), s!("World"), MB_OK);
        HookMessageBox.initialize(
            MessageBoxA,
            |_,_,_,_| { MessageBoxA(None, s!("Hooked"), s!("Hooked"), MB_OK) }
        ).unwrap().enable().unwrap();
        MessageBoxA(None, s!("Ansi"), s!("World"), MB_OK);
        HookMessageBox.disable().unwrap();
        MessageBoxA(None, s!("Ansi"), s!("World"), MB_OK);
    };
}

I have this test code, I get no error with the hook but it doesn't work as expected, the closure isn't executed (the normal call is done instead). So basically the test works, no error, but it doesn't work as expected. Is this normal ?

[Hpmason]

1 issue you may be running into is that the function signature of MessageBoxA from the windows crate is actually generic, so the actual function you're calling may be a completely different function than you're hooking.

In order to get the actual C function, you can get it's location in memory like we do in the messageboxw_detour example: https://github.com/Hpmason/retour-rs/blob/8dca3f2aa0ff3e4da486a6099e2c61e2c1c47c1b/examples/messageboxw_detour.rs#L51-L65

This makes sure you hook the proper C function that the windows crate calls.

Alternatively, if you just want to hook Windows functions you're calling, you can use the windows-sys crate which are raw bindings to the Windows libraries. But if you hook those, you're only hooking functions you call, since crates you use may use other versions of the windows or windows-sys crates. Also applications you may hook into will probably be using the windows C API directly, so you can't rely on hooking functions from the Windows crates

[Hpmason]

I was able to modify your example to have it work.

use retour::static_detour;
use windows::{s,core::PCSTR, Win32::UI::WindowsAndMessaging::*, Win32::Foundation::*};

static_detour! {
    static HookMessageBox: unsafe fn(HWND, PCSTR, PCSTR, MESSAGEBOX_STYLE) -> MESSAGEBOX_RESULT;
}

const MessageBoxAFn: unsafe fn(HWND, PCSTR, PCSTR, MESSAGEBOX_STYLE) -> MESSAGEBOX_RESULT = MessageBoxA::<HWND, PCSTR, PCSTR>;

fn main() {
    unsafe {
        MessageBoxAFn(HWND::default(), s!("Ansi"), s!("World"), MB_OK);
        HookMessageBox.initialize(
            MessageBoxAFn,
            |_,_,_,_| { MessageBoxA(None, s!("Hooked"), s!("Hooked"), MB_OK) }
        ).unwrap().enable().unwrap();

        MessageBoxAFn(HWND::default(), s!("Ansi"), s!("World"), MB_OK);
        HookMessageBox.disable().unwrap();
        MessageBoxAFn(HWND::default(), s!("Ansi"), s!("World"), MB_OK);
    };
}

Though, the same issues/restrictions from my previous comment still apply. This only guarantees that calls using MessageBoxAFn within your only library will be hooked since other code can use different versions of the windows crate

[4zv4l]

Thank you for the explanation and the code, I indeed try to hook another process calls. Long story short I made a clipboard monitoring tool in Nim to prevent users copy/pasting illegal content from a server during RDP Session and I wanted to rewrite it in Rust (so I can publish it and make it safer as well) and I need to be able to hook SetClipboardData and GetClipboardData. So I guess like you said I would have to use the raw C function definition ?

[4zv4l]

yeah I guess I'll have to do it this way

#![cfg(target_os = "windows")]

use retour::static_detour;
use windows::{s,core::PCSTR, Win32::UI::WindowsAndMessaging, Win32::Foundation::*};

extern "system" { fn MessageBoxA(hwnd: HWND, text: PCSTR, title: PCSTR, style: u32) -> i32; }

static_detour! {
    static HookMessageBox: unsafe extern "system" fn(HWND, PCSTR, PCSTR, u32) -> i32;
}

fn hookedMessageBox(hwnd: HWND, text: PCSTR, _caption: PCSTR, msgbox_style: u32) -> i32 {
    unsafe { HookMessageBox.call(hwnd, text, s!("Detoured"), msgbox_style) }
}

fn main() {
    unsafe {
        WindowsAndMessaging::MessageBoxA(None, s!("Ansi"), s!("World"), WindowsAndMessaging::MB_OK);
        HookMessageBox.initialize(MessageBoxA, hookedMessageBox).unwrap().enable().unwrap();
        WindowsAndMessaging::MessageBoxA(None, s!("Ansi"), s!("World"), WindowsAndMessaging::MB_OK);
        HookMessageBox.disable().unwrap();
        WindowsAndMessaging::MessageBoxA(None, s!("Ansi"), s!("World"), WindowsAndMessaging::MB_OK);
    };
}

For the tests I can use the windows crate, I still don't understand what you meant about the windows create function being generic, if I call MessageBox using the s!() it should be 'obvious' that it will call MessageBoxA right ?

[4zv4l]

well just tried injecting a dll with this and it doesn't work . I really need to use get_module_symbol_address("user32.dll", "MessageBoxA") ? using the extern isn't enough ?

[4zv4l]

indeed with your get_module_symbol_address("user32.dll", "MessageBoxA") it works great

[Hpmason]

For the tests I can use the windows crate, I still don't understand what you meant about the windows create function being generic, if I call MessageBox using the s!() it should be 'obvious' that it will call MessageBoxA right ?

If you look at the windows crate definition of MessageBoxA, the first 3 parameters are generic. For any uniquely typed version of the function your try to call, Rust will generate an entire function for it in the final executable of your program. Here's a godbolt example. If you look at the assembly output, there's 2 versions of the double function with a bunch of "weird" symbols which represent the compiled versions of the generic function (_ZN7example6double17h4d7ba8b15267fc9dE for f32 and _ZN7example6double17ha4e5249737a77dd0E for i32)

So, in your original example you call MessageBoxA(None, s!("Ansi"), s!("World"), WindowsAndMessaging::MB_OK);, the type of which is unsafe fn(Option<HWND>, PCSTR, PCSTR, MESSAGEBOX_STYLE) -> MESSAGEBOX_RESULT which has it's own function in the final executable. Your hook hooks a function of type unsafe extern "system" fn(HWND, PCSTR, PCSTR, u32) -> i32 which also gets it's own function in the final executable. When you initialize your hook (HookMessageBox) you are hooking that the second function that exists in memory.

It's really hard to hook a generic function because each variant of it has it's own copy in your program.

[4zv4l]

Alright I understand, I'm struggling trying to rewrite an project I made in Nim in Rust, I thought I got the hooking part right but it seems more complex than what I had to deal with Nim (different library and slightly different way to deal with the hook). Don't wanna do advertising but seems you look to know your stuff, if you wanna check https://github.com/4zv4l/ClipMon. I'll close this issue since I understand your explanation and got it to work with a dummy program. Thanks ^^

[Hpmason]

Glad I could help and I always like hearing what people are working on!