Bug report: Presence of Live++ prevents OBS graphics hook from capturing application

[echolox]

Scenario:

I'm developing a game that uses D3D11 and DXGI for rendering, the swap-chain etc. When trying to capture the application via Open Broadcaster Software (OBS) Studio (version 30.0.2) it is not possible to capture the application using the "Game Capture" source. Only a "Window Capture" or full "Display Capture" works.

This problem is not critical, since it's relatively trivial to disable Live++ when I intend to capture game footage, which I'd do on a release build anyway. But it has tripped me up for a while that "Game Capture" wasn't working. I spent a lot of time looking through the setup code for my renderer trying things out until I stumbled upon the debug output below and tried the application without Live++ running. Ultimately my worry was just that players of the game would eventually run into the same problem trying to capture the game using "Game Capture". I'm no longer worried about that now :)

I don't understand enough about the details to say whether Live++ or OBS is in a position to fix this issue, or whether it's just an inherent limitation related to Microsoft Detours that cannot be solved. So maybe it would be enough to add this in the Documentation or FAQ so it's easier to figure out for other people in the future.

Steps to reproduce:

1. Open the application and OBS (the order does not matter)
2. Create a new "Game Capture" source in any scene in OBS
3. Select the application using the default settings (changing any of them doesn't seem to help anyway) and click OK
4. The preview window in OBS doesn't show the application

For comparison, creating a "Window Capture" in the same way works.

Thankfully OBS prints to the debug output when it attempts to hook into an application. When it fails the output looks as follows:

[OBS] graphics-hook.dll loaded against process: <application name>.exe
[OBS] (half life scientist) everything..  seems to be in order
[OBS] Failed to attach Detours hook: 8
[OBS] Failed to attach Detours hook: 8
[OBS] Failed to attach Detours hook: 8
.... repeating that last line forever

When it works it looks like this:

[OBS] graphics-hook.dll loaded against process: <application name>.exe
[OBS] (half life scientist) everything..  seems to be in order
[OBS] Hooked IDXGISwapChain::Present
[OBS] Hooked IDXGISwapChain::ResizeBuffers
[OBS] Hooked IDXGISwapChain1::Present1
[OBS] Hooked IDXGISwapChain::Release
[OBS] Hooked DXGI
[OBS] Found D3D11 11.0 device on swap chain
[OBS] DXGI_SWAP_CHAIN_DESC:
    BufferDesc.Width: 1920
    BufferDesc.Height: 1080
    BufferDesc.RefreshRate.Numerator: 0
    BufferDesc.RefreshRate.Denominator: 0
    BufferDesc.Format: 87
    BufferDesc.ScanlineOrdering: 0
    BufferDesc.Scaling: 0
    SampleDesc.Count: 1
    SampleDesc.Quality: 0
    BufferUsage: 32
    BufferCount: 2
    Windowed: 1
    SwapEffect: 4
    Flags: 64
[OBS] d3d11 shared texture capture successful

Diving a little into the OBS source code the failure likely occurs in dxgi-capture.cpp in the function hook_dxgi: https://github.com/obsproject/obs-studio/blob/master/plugins/win-capture/graphics-hook/dxgi-capture.cpp

DetourTransactionBegin();

RealPresent = (present_t)present_addr;
DetourAttach(&(PVOID &)RealPresent, hook_present);

RealResizeBuffers = (resize_buffers_t)resize_addr;
DetourAttach(&(PVOID &)RealResizeBuffers, hook_resize_buffers);

if (present1_addr) {
    RealPresent1 = (present1_t)present1_addr;
    DetourAttach(&(PVOID &)RealPresent1, hook_present1);
}

const LONG error = DetourTransactionCommit();

That last error receives the value 8 which matches "0x00000008 - ERROR_NOT_ENOUGH_MEMORY" from this table: https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-erref/18d8fbe8-a967-4f1c-ae50-99ca8e491d2d Details on DetourAttach: https://github.com/microsoft/Detours/wiki/DetourAttach

[MolecularMatters]

Thanks for the report.

I looked into this to understand what's going on. When registering a module with Live++, it scans the surrounding +-2GB memory range/virtual address space and reserves free pages. When loading and installing a patch for a module later on, these free pages are freed temporarily, and the OS is tricked into loading the patch DLL inside that range. Afterwards, the remaining pages are reserved again. This is required since patches need to be loaded in the +-2GB address range in order to patch relocations to existing symbols.

However, this means that OBS is unable to hook the DXGI functions. Similar to Live++, Detours needs to be able to allocate some space for the trampoline next to the original function. But since all pages surrounding that function are reserved, this allocation fails.

I guess you're using LPP_MODULES_OPTION_ALL_IMPORT_MODULES when enabling modules? If your application is a single executable, use LPP_MODULES_OPTION_NONE instead. This will leave all other DLLs alone and not reserve any pages surrounding them, which should make Detours/OBS work.

[echolox]

Thank you for looking into it. Your suggestion was on point!

I've now switched from using LPP_MODULES_OPTION_ALL_IMPORT_MODULES to enabling the modules I need individually (the main .exe and the .dll for the renderer) using LPP_MODULES_OPTION_NONE. OBS hooked into the DXGI functions successfully.

Thank you so much for the quick support and for this invaluable tool in general.