veeenu / hudhook

A videogame overlay framework written in Rust, supporting DirectX and OpenGL
MIT License
209 stars 30 forks source link

Unified renderer #143

Closed veeenu closed 10 months ago

veeenu commented 10 months ago

This is a MASSIVE refactor that completely changes the underlying rendering engine, which is now a unified DirectX12 renderer that is then composited over games' windows via DirectComposition, with barely any public API change.

It seamlessly works with Dark Souls III and Elden Ring already, and the integration test cases.

Being such a breaking change, I will shamelessly ping all people that may be affected as I don't have a lot of control over corner cases but really don't want to merge this in before I'm sure I'm not disrupting anything. I would be super grateful if you'd be up for checking this out. I expect some of you may have moved on to other projects, in that case I hope you won't be annoyed by the ping!

Explanation

This PR implements a DirectX 12 renderer in hudhook::renderer::dx12 which renders over an off-screen surface, and then uses DirectComposition to composite the off-screen surface on top of the HWND, completely sidestepping the games' native rendering engine which will run on its own.

Present hooks now only serve the purpose of actually triggering a render and a composition and the device objects aren't touched at all. In fact, most of the hooks' rendering logic except building the actual hooked function implementations is now shared and identical across all hooks.

As a result, hooks are now tiny compared to what they were before and their maintenance is going to be very easy. Similarly, with a single renderer there is much less code to maintain.

Additionally, this will make it extremely simple to add new renderers

Affected issues

As a bonus, this PR should fix the following issues, or at the very least give us a cleaner path to their resolution.

If you are getting pinged, it would mean the world if you could give this branch a spin and report back if it fixed the associated issue.

Thank you all for your patience and sorry again for the ping! 😅

camas commented 10 months ago

Unfortunately #79 still an issue. Tested with 7791c49

Fails soon after injection when it can't lock the render engine, with a bunch of WndProc lock fails before.

19:00:12 [DEBUG] (1) hudhook::hooks: WNDCLASSEXW { cbSize: 80, style: WNDCLASS_STYLES(3), cbClsExtra: 0, cbWndExtra: 0, hInstance: HINSTANCE(140696328601600), hIcon: HICON(0), hCursor: HCURSOR(0), hbrBackground: HBRUSH(0), lpszMenuName: PCWSTR(0x0), lpszClassName: PCWSTR(0x7ffbf3eff8e8), hIconSm: HICON(0) }
19:00:12 [DEBUG] (1) hudhook::hooks: HWND(788424)
19:00:12 [INFO] Output window: IDXGISwapChain(IUnknown(0x22b60745720))
19:00:12 [INFO] Desc: DXGI_SWAP_CHAIN_DESC { BufferDesc: DXGI_MODE_DESC { Width: 1920, Height: 1080, RefreshRate: DXGI_RATIONAL { Numerator: 0, Denominator: 1 }, Format: DXGI_FORMAT(28), ScanlineOrdering: DXGI_MODE_SCANLINE_ORDER(0), Scaling: DXGI_MODE_SCALING(0) }, SampleDesc: DXGI_SAMPLE_DESC { Count: 1, Quality: 0 }, BufferUsage: DXGI_USAGE(48), BufferCount: 2, OutputWindow: HWND(1771696), Windowed: BOOL(1), SwapEffect: DXGI_SWAP_EFFECT(3), Flags: 2114 }
19:00:25 [DEBUG] (2) hudhook::hooks::render: Could not lock in WndProc
...
19:01:01 [DEBUG] (2) hudhook::hooks::render: Could not lock in WndProc
19:01:01 [ERROR] Could not lock render engine
veeenu commented 10 months ago

@camas I stalked your GitHub and noticed you are using hudhook for Golf With Your Friends. I have that game so I can take a look at it directly! But yeah I can easily see how that's a synchronization issue. It could be solved by using threads and channels but I'm not sure I want the added overhead of those.

Weirdly enough, there is only 3 places now where the render engine is locked that could create contention:

My hunch there is that the wnd proc calls itself recursively, locks once and is not ever able to exit and do anything else. Or the WND_PROC static is initialized twice (which shouldn't happen as we use get_or_init that has the guarantee of uniqueness) and thus called recursively via CallWindowProcW.

veeenu commented 10 months ago

I have gotten some feedback and will now merge this in as there are no apparent regressions for the time being.

vars1ty commented 10 months ago

Thanks for the ping, feel free to ping me whenever as I'm more than happy to test out features whenever I can.

I'll update to the latest commit, fix a few errors due to the changes and then test it in both OpenGL and then later DirectX9, will let you know how it goes 👍🏽

veeenu commented 10 months ago

Thank you 🙏