search

obsproject / obs-studio

OBS Studio - Free and open source software for live streaming and screen recording
https://obsproject.com
GNU General Public License v2.0
60.55k stars 8.01k forks source link

Graphics-hook.dll crashes openGL program (SwapBuffers using dxgi on windows?) #9168

Open Reabs opened 1 year ago

Reabs commented 1 year ago

Operating System Info

Windows 10

Other OS

No response

OBS Studio Version

29.1.3

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/QbSPUOoZFrMFhQda

OBS Studio Crash Log URL

No response

Expected Behavior

My program would not crash when disabling and re-enabling capture source in obs.

Current Behavior

Disabling and re-enabling capture feed in obs will crash my program using openGL due to an access violation inside of graphics-hook.dll

Steps to Reproduce

  1. Add game capture for openGL program
  2. Let game feed be captured normally
  3. Disable source
  4. Re-enable source
  5. Program crashes

Anything else we should know?

I use the same engine for a lot of different programs and they all have this same issue I've tried to see if there is some way I can change how swapbuffers is working but google doesn't seem to be any help and the feature on nvidia's control panel "Vulkan/OpenGL present method" doesn't seem to have any effect on it.

I believe this is related to this issue https://github.com/obsproject/obs-studio/pull/970

I think this is because SwapBuffers seems to use dxgi under the hood. If I remove SwapBuffers from my code when obs tries to hook into my program it outputs this....

 [OBS] graphics-hook.dll loaded against process: GameSync.exe
 [OBS] (half life scientist) everything..  seems to be in order
 [OBS] Hooked SwapBuffers
 [OBS] Hooked wglDeleteContext
 [OBS] Hooked wglSwapLayerBuffers
 [OBS] Hooked wglSwapBuffers
 [OBS] Hooked GL

however when I make the call to SwapBuffers then the output looks like this....

 [OBS] (half life scientist) everything..  seems to be in order
 [OBS] Hooked IDXGISwapChain::Present
 [OBS] Hooked IDXGISwapChain::ResizeBuffers
 [OBS] Hooked IDXGISwapChain1::Present1
 [OBS] Hooked IDXGISwapChain::Release
 [OBS] Hooked DXGI
 [OBS] Hooked SwapBuffers
 [OBS] Hooked wglDeleteContext
 [OBS] Hooked wglSwapLayerBuffers
 [OBS] Hooked wglSwapBuffers
 [OBS] Hooked GL
 [OBS] Shared-texture OpenGL capture available
 'GameSync.exe' (Win32): Loaded 'C:\Windows\System32\DriverStore\FileRepository\nvmii.inf_amd64_bbbaf2edf0a69617\NvCamera\NvCameraAllowlisting32.dll'. 
 'GameSync.exe' (Win32): Unloaded 'C:\Windows\System32\DriverStore\FileRepository\nvmii.inf_amd64_bbbaf2edf0a69617\NvCamera\NvCameraAllowlisting32.dll'
 [OBS] gl shared texture capture successful

This is actually still fine as long as the source remains active at all times however if I disable the source or switch to a scene that doesn't have this source obs does this....

 [OBS] ------------------ gl capture freed ------------------
 [OBS] Found D3D11 11.0 device on swap chain

Which then when I either re-enable the source or switch back to the scene with the source in it my program will crash inside of graphics-hook.dll with this "Exception thrown at 0x77CCF633 (ntdll.dll) in GameSync.exe: 0xC0000005: Access violation writing location 0x00000014."

Call Stack:

 ntdll.dll!RtlpWaitOnCriticalSection()
 ntdll.dll!RtlpEnterCriticalSectionContended()
 ntdll.dll!_RtlEnterCriticalSection@4()
 graphics-hook32.dll!7c8f23f0()
 graphics-hook32.dll![Frames below may be incorrect and/or missing, no symbols loaded for graphics-hook32.dll]  
 [External Code]

Occasionally one other scenario can happen where it seems to actually hook to the dxgi (Maybe 1 out or 20 executions)....

 [OBS] graphics-hook.dll loaded against process: GameSync.exe
 [OBS] (half life scientist) everything..  seems to be in order
 [OBS] Hooked IDXGISwapChain::Present
 [OBS] Hooked IDXGISwapChain::ResizeBuffers
 [OBS] Hooked IDXGISwapChain1::Present1
 [OBS] Hooked IDXGISwapChain::Release
 [OBS] Hooked DXGI
 [OBS] Hooked SwapBuffers
 [OBS] Hooked wglDeleteContext
 [OBS] Hooked wglSwapLayerBuffers
 [OBS] Hooked wglSwapBuffers
 [OBS] Hooked GL
 [OBS] Found D3D11 11.0 device on swap chain
 [OBS] DXGI_SWAP_CHAIN_DESC:
     BufferDesc.Width: 768
     BufferDesc.Height: 320
     BufferDesc.RefreshRate.Numerator: 0
     BufferDesc.RefreshRate.Denominator: 1
     BufferDesc.Format: 87
     BufferDesc.ScanlineOrdering: 0
     BufferDesc.Scaling: 0
     SampleDesc.Count: 1
     SampleDesc.Quality: 0
     BufferUsage: 96
     BufferCount: 3
     Windowed: 1
     SwapEffect: 3
     Flags: 2050
 [OBS] d3d11 shared texture capture successful
 [OBS] Shared-texture OpenGL capture available

When this happens everything seems to work as normal and disabling and re-enabling the source over and over seems to have no problem...

 [OBS] ------------------ gl capture freed ------------------
 'GameSync.exe' (Win32): Loaded 'C:\Windows\System32\DriverStore\FileRepository\nvmii.inf_amd64_bbbaf2edf0a69617\NvCamera\NvCameraAllowlisting32.dll'. 
 'GameSync.exe' (Win32): Unloaded 'C:\Windows\System32\DriverStore\FileRepository\nvmii.inf_amd64_bbbaf2edf0a69617\NvCamera\NvCameraAllowlisting32.dll'
 [OBS] gl shared texture capture successful
jpark37 commented 1 year ago

Feels like a variation of #6698 but OpenGL instead of Vulkan. Can probably be fixed the same way if someone is interested.

Lain-B commented 1 year ago

We spent time debugging this issue, and here are the two three main points we discovered:

  1. NVIDIA's driver/interface is creating its own DXGI context separately from the OpenGL context
  2. The rendering happens in a completely separate thread
  3. It only appears to happen with laptops (presumably because of multiple adapters)

My personal theory is that for laptops, NVIDIA's internal code creates a separate thread for blitting the OpenGL backbuffer to the desktop or something. Either way, it happens in a separate thread than the OpenGL thread of the process, so it's possible that the passthrough trick may not be adequate.

Reabs commented 1 year ago

So I took a stab at fixing this doing the methods we talked about but it seems like it will be a bit more complicated then just add a variable for first hooked and then don't let it hook into the other systems. After doing the fix and running my current setup I can avoid the crash.

Testing with various games however leads to various problems where games will sometimes hook into different setups that they don't use anyway notably hooking into dx9 first but then the capture hook you want is actually through dxgi. Also I've run into other games that will hook into openGL even though they end up using dx. My own setup seems to not work unless I have the openGL hook before the directX hook (I have multiple game captures of different programs at once) which is why I was able to notice the previous issue.

Just wanted to give an update to this since I said I would try to do a pull but also don't want to cause other issues to surface.

BluSpring commented 1 year ago

We spent time debugging this issue, and here are the ~two~ three main points we discovered:

  1. NVIDIA's driver/interface is creating its own DXGI context separately from the OpenGL context
  2. The rendering happens in a completely separate thread
  3. It only appears to happen with laptops (presumably because of multiple adapters)

My personal theory is that for laptops, NVIDIA's internal code creates a separate thread for blitting the OpenGL backbuffer to the desktop or something. Either way, it happens in a separate thread than the OpenGL thread of the process, so it's possible that the passthrough trick may not be adequate.

This might've been the result of NVIDIA's "Threaded Optimizations", which has been known to cause several problems with OpenGL (a quick example is the Sodium mod for Minecraft).

Squall-Leonhart commented 4 months ago

This might've been the result of NVIDIA's "Threaded Optimizations", which has been known to cause several problems with OpenGL (a quick example is the Sodium mod for Minecraft).

Not relevant.

This is probably due to combined hooking from multiple applications, which OBS doesn't perform well via its stock implementation of Detours.

Nahimic, Default install of RTSS (has a detours compatible mode at the users discretion) Asus GPU tweak overlay, etc, can all die when trying to co-exist with OBS leading to crashing.