obsproject / obs-studio

OBS Studio - Free and open source software for live streaming and screen recording
https://obsproject.com
GNU General Public License v2.0
59.55k stars 7.89k forks source link

Segfault after destroying a window during injection graphics-hook.dll on Vulkan apps #9605

Closed wusikijeronii closed 1 year ago

wusikijeronii commented 1 year ago

Operating System Info

Windows 11

Other OS

No response

OBS Studio Version

29.1.3

OBS Studio Version (Other)

No response

OBS Studio Log URL

Don't matter

OBS Studio Crash Log URL

No response

Expected Behavior

Hello. I am not an OBS Studio user. I am a member of the development team of our own GUI application. I expect not to get segfault errors when our application is not related to OBS Studio in any way.

Current Behavior

While developing and testing a Vulkan-based application on an NVIDIA GPU, I encountered the problem of destroying a window in the presence of an injection of graphics-hook.dll. This file, as I understand it, is related to OBS. The GLFW3 library is used to create windows. Tested GPU:

  1. Nvidia RTX 2070
  2. Nvidia RTX 3050 Laptop
  3. AMD Graphics

Details:

  1. The problem only appears on NVIDIA GPUs when using Vulkan.
  2. The error occurs when trying to destroy a window via GLFW.
  3. The problem has been noticed in various applications using Vulkan.

Temporary solution: adding a delay before destroying the window prevents the error. C++ example:

Window::~Window()
{
    logInfo("backend", "Destroying GLFW window %p", _glfwWindow);
    std::this_thread::sleep_for(std::chrono::milliseconds(100));
    glfwDestroyWindow(_glfwWindow);
}

It looks like the obs dll doesn't have time to destroy resources

Debug details

Before launch gdb output:

warning: [OBS]
warning: graphics-hook.dll loaded against process: app3d.exe
warning:
warning: [OBS]
warning: (half life scientist) everything..  seems to be in order
warning:
WinDBG backtrace: # RetAddr Args to Child Call Site
00 00007ffd2d3242c2 0000000000000001 00007ffcb72c49c9 0000021c08eb7160 00007ffd2dd19570 <Unloaded_nvoglv64.dll>+0x1384260
01 00007ffd2d324620 0000021c0c2a00f0 00007ff79f553dac 00007ff79f8091a0 00007ff79f5a0ad7 USER32!DispatchHookA+0x1c2
02 00007ffd2d31302d 0000000000000000 0000000000000000 0000021c08e80000 0000000000000000 USER32!fnHkINLPCWPSTRUCTA+0x60
03 00007ffd2dd52d94 0000021c00000000 00000046c0fad930 00000000000a0008 00007ffcb8320c02 ntdll!KiUserCallbackDispatcherContinue
04 00007ffd2b0e2754 00007ffcb82fe807 0000000000000001 0000021c0c28ce50 00000046c0fadb80 win32u!NtUserDestroyWindow+0x14
05 00007ffcb82fe807 0000000000000001 0000021c0c28ce50 00000046c0fadb80 0000000000000000 glfw3!glfwGetWin32Monitor+0x2107
06 00007ffcb82f796c 0000000000000001 0000021c08eb7160 00000046c0fadb98 00000046c0fada50 glfw3!glfwDestroyWindow+0x8c
07 00007ff79f59b3ce 00000046c0fadbb0 fab64d0000000000 796f72747365442a 535f4b5620676e69 0x4b3ce
08 00007ff79f7721a8 0000021c00000000 0000000000000000 0000000000000000 c040000040400000 0x2221a8
09 00007ff79f78288b 0000000000000001 0000021c00000000 0000000000000038 0000021c00000070 0x23288b
10 00007ff79f551315 0000000000000001 00007ff79f809368 00007ff79f809360 0000000000000000 0x1315
11 00007ff79f551366 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0x1366
12 00007ffd2d56257d 0000000000000000 0000000000000000 0000000000000000 0000000000000000 KERNEL32!BaseThreadInitThunk+0x1d
13 00007ffd2dd0aa68 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ntdll!RtlUserThreadStart+0x28

I would be grateful for consideration of this problem. If additional information or details are required for playback, please let me know.

Steps to Reproduce

You can use any Vulkan example. For tests I use this repo.

  1. Build the project
  2. Use your prefer debugger 'cos without debug attaching the issue is periodic For example, if you use gdb run:
    gdb [myapp]
    run

Anything else we should know?

After removing the OBS Studio the problem was solved

wusikijeronii commented 1 year ago

My bad. I was wrong. The problem does not apply to you.