[Linux] Fail to hook HL2

[Edremon]

I'm trying to use your overlay on HL2. It fails to hook be it with the OpenGL renderer or the Vulkan one. I'm getting the following log:

Vulkan

[trace] - operator(){463} - Started renderer detection.
[info] - _HookVulkan{362} - Hooked vkQueuePresentKHR to detect Vulkan
[error] - _HookVulkan{379} - Failed to hook _VkQueuePresentKHR
[info] - ~VulkanHook_t{1066} - VulkanHook_t Hook removed
[trace] - operator(){519} - Renderer detection done 0x0.

OpenGL

[trace] - operator(){463} - started renderer detection.
[info] - _hookopenglx{335} - hooked glxswapbuffers to detect openglx
[error] - _hookopenglx{345} - failed to hook _glxswapbuffers
[info] - ~openglxhook_t{188} - openglx hook removed
[trace] - operator(){519} - renderer detection done 0x0.

Note that I tried the linux_opengl_app and linux_vulkan_app test and overlay on both are mostly working. I don't see any issues on the OpenGL one, with Vulkan one, rendering get fully black once hooked (only visible thing is overlay after hitting SHIFT+F2), without hook, vulkan example have standard imgui demo.

[Nemirtingas]

Renderer detector failed to hook opengl and vulkan functions. It is expected that the overlay doesn't work in that case.

[Edremon]

I do understand that it's failing to hook those functions. What I don't understand is why.

I tested a bit more and somehow, on both OpenGL and Vulkan, 10-20% of the time it manages to hook but keybinds to open overlay doesn't work (so, I don't know if it's really working). Also, with Vulkan after a few seconds, everything turn black like in test vulkan app and during hooking step, some of the tries had this error: [error] - StartHook{82} - Failed to hook VkCreateSwapchainKHR but even without it, still turned black.

[Nemirtingas]

For keybinds, see #22 . To further debug the hook failure, you must debug mini_detour. Functions might not have enought relocatable code to create a trampoline, as such, the hook fails.

[Edremon]

I'm not sure if the keybinds problem is the same, putting same trace line and never triggered, the only log I get about X11 is StartHook{179} - Hooked X11. As for failing to hook, I don't think I have enough knowledge to debug mini_detour.

[Nemirtingas]

To debug mini_detour, it's not well defined, but you can define USE_SPDLOG : https://github.com/Nemirtingas/mini_detour/blob/b66ae42406f47d79527d4dd73f73f71d55aefa5b/src/mini_detour.cpp#L14

Logs will be enabled after that

[Edremon]

Didn't manage to get any SPD output, but with gdb, I see that it fail here: https://github.com/Nemirtingas/mini_detour/blob/b66ae42406f47d79527d4dd73f73f71d55aefa5b/src/mini_detour.cpp#L891

[Nemirtingas]

That means memory couldn't be allocated near the function you want to hook.

[Nemirtingas]

Any Linux development is stale until #22 is solved. Might also need a Wayland hook.