Allow DirectX hooking using existing device pointer

[earthnuker]

Hi,

would it be possible to add a function to initialize hudhook with an existing Direct9Device pointer? one of my projects is an overlay for a DirectX8 game which i'm running through d3d8to9 but initialize hudhook using the following code:

// imports, Overlay struct definition and empty ImguiRenderLoop impl left out for brevity
static OVERLAY_INITIALIZED: AtomicBool = AtomicBool::new(false);
pub fn init() -> Result<()> {
    if OVERLAY_INITIALIZED.swap(true, Ordering::SeqCst) {
        bail!("Tried to initialize overlay twice!");
    }
    // Passing a handle to the module around into the function would be a bit cumbersome, hence GetModuleHandleA()
    let hmodule = unsafe {std::mem::transmute(windows::Win32::System::LibraryLoader::GetModuleHandleA(None)?)};
    hudhook::lifecycle::global_state::set_module(hmodule);
    let hooks: Box<dyn hooks::Hooks> = Overlay::new().into_hook::<ImguiDx9Hooks>();
    unsafe {hooks.hook()};
    hudhook::lifecycle::global_state::set_hooks(hooks);
    println!("OK!");
    Ok(())
}

fails with the following error message:

panicked at 'IDirect3DDevice9::CreateDevice: failed to create device: Error { code: 0x8876086C, message:  }', C:\Users\Earthnuker\scoop\persist\rustup-msvc\.cargo\registry\src\index.crates.io-6f17d22bba15001f\hudhook-0.4.0\src\hooks\dx9.rs:324:6

Best regards,

Earthnuker

[veeenu]

Hi!

This would probably require a big change in the API, though I admit if we had access to devices before having the hooks we could skip the whole song and dance of initializing dummy devices for finding out the Present address. Which would be pretty good.

That panic doesn't depend on the device that's found in the game, though; rather in a failure to create a new dummy device with DirectX 9.

I don't know about the inner workings of d3d8to9 but it feels like it could be one of these two:

• under the hood the game does Dx8 calls that get routed to Dx9 calls (and hence a DirectX 9 device is available somewhere)
• the game has Dx9 calls available that actually get routed to Dx8 internally (and hence we would need a DirectX 8 specific hook).

I've never worked with older DirectX releases (9 included, I didn't create it) so I'm not sure how to proceed in this case. I feel like it would be worth it investigating the exact reasons for that panic, though. Unfortunately observability is a bit lacking in this case so that might be tricky. By the way, what game is it? I can try running hudhook against it if I happen to have it, maybe I can get to an answer faster.

Feel free to open a PR for this if you have ideas about how to proceed!

[earthnuker]

Hi,

Thanks for the quick response!

the error code appears to be D3DERR_INVALIDCALL although i'm not sure why CreateDevice() would return that error as the CreateDevice call here looks sensible to me.

as for the way d3d8to9 works, it's the former, d3d8to9 works by providing its own CreateDevice that returns a proxy device with a virtual method table that intercepts functions and (if applicable) modifies their parameters and forwards them to a Direct3DDevice9. currently i find the Direct3DDevice9 pointer via pattern scanning to find the global variable containing the device pointer and the offset at which it is stored, plus a hard-coded offset to get from the Direct3DDevice8 used by the game to the Direct3DDevice9 used by d3d8to9, that way i could also call my own functions with the device initialized by the game.

the game i'm testing against is Scrapland Remastered the original is from 2005 which is probably why they still use DirectX8 (and maybe MercurySteam recycles some internal game engine for it)

Best regards, Earthnuker

[veeenu]

the error code appears to be D3DERR_INVALIDCALL although i'm not sure why CreateDevice() would return that error as the CreateDevice call here looks sensible to me.

This has happened often before -- seemingly correct CreateDevice calls will error out as invalid and we don't have a good way of determining why that happens.

as for the way d3d8to9 works, it's the former, d3d8to9 works by providing its own CreateDevice that returns a proxy device with a virtual method table that intercepts functions and (if applicable) modifies their parameters and forwards them to a Direct3DDevice9.

Ok, so de facto the DirectX 9 calls are all that really happens under the hood, so the dx9 hook should work.

currently i find the Direct3DDevice9 pointer via pattern scanning to find the global variable containing the device pointer and the offset at which it is stored, plus a hard-coded offset to get from the Direct3DDevice8 used by the game to the Direct3DDevice9 used by d3d8to9, that way i could also call my own functions with the device initialized by the game.

Hmmm, this is tricky. Definitely not generalizable. But on the other hand you should not need to inject your own device at all; hooking the Present function should be more than enough to already get that where it's needed. The reason we create a (dummy) device there is only to find the address of the Present function in the vtable; if we manage to have that not crash, chances are the rest of the code will work just fine.

I need to add some observability to that -- currently the way I handle errors is very crude, but there are issues that plan to get that sorted out.

the game i'm testing against is Scrapland Remastered the original is from 2005 which is probably why they still use DirectX8 (and maybe MercurySteam recycles some internal game engine for it)

I don't have access to that unfortunately. Best I can do is add observability so that the errors can be a little easier to understand.

[SkyLeite]

I ran into this same issue, but without d3d8to9. The issue is (I think) related to DummyHwnd in get_dx9_present_addr. When I use hudhook, right before crashing, Wine spits out 010c:fixme:vulkan:X11DRV_vkCreateWin32SurfaceKHR Application requires child window rendering, which is not implemented yet!. I understand this is Wine's limitation, but having an API for passing the address of the DX9 functions directly would alleviate the issue.

[veeenu]

Mmm, if it is a limitation of Wine there's not much to do there. It's weird because afaik some stuff made in hudhook does work unmodified on the steam deck. Could try going back to using GetDesktopWindow to get a handle but I've lost count of the times we went back and forth due to compatibility issues.

On the other hand, the new unified rendering approach as of #141 might remove the need for an external window altogether as I've figured out setting up the swap chains to an off screen surface to get the vtable pointers without attaching a hwnd. That might be applicable to dx9 too, not sure but hopeful.

[SkyLeite]

Yeah, I wouldn't expect the library to work around Wine limitations for sure. Just wanted to point that out in case other people run into the same issue. I currently have it working with a vendored copy of src/hooks/dx9.rs and manually setting the address of the structure.

As for why it works on the Steam Deck, I would imagine that this is something Proton has a fix for. To be honest it caught me off guard to see that error message because it seems like such a fundamental function for Wine to handle!

[veeenu]

The issue is (I think) related to DummyHwnd in get_dx9_present_addr. When I use hudhook, right before crashing, Wine spits out 010c:fixme:vulkan:X11DRV_vkCreateWin32SurfaceKHR Application requires child window rendering, which is not implemented yet!

I may have found a workaround for this; tried to make DummyHwnd not be a child window. Currently, it is a child of HWND_MESSAGE which you do when you don't want to actually use the hwnd for anything other than a message loop (like we do), but it is a child indeed. Removing the parent altogether should solve this issue.

The bad news is this will only land when the porting as of #141 will be done. The good news is that it might happen soonish.