renderdoccmd recursively calls glXGetProcAddress

djdeath commented 5 years ago

Description

Trying to capture a game frame, I'm launching the game with :

renderdoccmd capture /path/to/game

The command crashes with an endless backtrace of glXGetProcAddress (trying with MadMax from steam) :

#149616 0x00007f32c94a6e20 in glXGetProcAddress_renderdoc_hooked (f=0x7f32c9e73ef6 "glXCreateContextAttribsARB") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:497
#149617 0x00007f32c94a7191 in glXGetProcAddress (f=0x7f32c9e73ef6 "glXCreateContextAttribsARB") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:592
#149618 0x00007f32c94a6e20 in glXGetProcAddress_renderdoc_hooked (f=0x7f32c9e73ef6 "glXCreateContextAttribsARB") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:497
#149619 0x00007f32c94a7191 in glXGetProcAddress (f=0x7f32c9e73ef6 "glXCreateContextAttribsARB") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:592
#149620 0x00007f32c94a6e20 in glXGetProcAddress_renderdoc_hooked (f=0x7f32c9e73ef6 "glXCreateContextAttribsARB") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:497
#149621 0x00007f32c94a7009 in glXGetProcAddressARB_renderdoc_hooked (f=0x7f32c9e73ef6 "glXCreateContextAttribsARB") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:536
#149622 0x00007f32c94a71ab in glXGetProcAddressARB (f=0x7f32c9e73ef6 "glXCreateContextAttribsARB") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:597
#149623 0x00007f32c94a813f in GLXHooked (handle=0x7f32c9036050) at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:724
#149624 0x00007f32c94a20d4 in std::_Function_handler<void (void*), void (*)(void*)>::_M_invoke(std::_Any_data const&, void*&&) (__functor=..., __args#0=@0x7fff09e7eb20: 0x7f32c9036050) at /usr/include/c++/8/bits/std_function.h:297
#149625 0x00007f32c9dee965 in std::function<void (void*)>::operator()(void*) const (this=0x7fff09e7eb70, __args#0=0x7f32c9036050) at /usr/include/c++/8/bits/std_function.h:687
#149626 0x00007f32c9dedc13 in CheckLoadedLibraries () at /home/djdeath/src/mesa-src/renderdoc/renderdoc/os/posix/linux/linux_hook.cpp:119
#149627 0x00007f32c9dee109 in LibraryHooks::EndHookRegistration () at /home/djdeath/src/mesa-src/renderdoc/renderdoc/os/posix/linux/linux_hook.cpp:189
#149628 0x00007f32c9b2cd24 in LibraryHooks::RegisterHooks () at /home/djdeath/src/mesa-src/renderdoc/renderdoc/hooks/hooks.cpp:47
#149629 0x00007f32c9e01609 in library_loaded () at /home/djdeath/src/mesa-src/renderdoc/renderdoc/os/posix/posix_libentry.cpp:71
#149630 0x00007f32c9e01763 in init::init (this=0x7f32ca333cd0 <do_init>) at /home/djdeath/src/mesa-src/renderdoc/renderdoc/os/posix/posix_libentry.cpp:84
#149631 0x00007f32c9e01678 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at /home/djdeath/src/mesa-src/renderdoc/renderdoc/os/posix/posix_libentry.cpp:85
#149632 0x00007f32c9e0168e in _GLOBAL__sub_I_posix_libentry.cpp(void) () at /home/djdeath/src/mesa-src/renderdoc/renderdoc/os/posix/posix_libentry.cpp:92
#149633 0x00007f32ca5810ca in call_init (l=<optimized out>, argc=argc@entry=2, argv=argv@entry=0x7fff09e7eda8, env=env@entry=0x7fff09e7edc0) at dl-init.c:72
#149634 0x00007f32ca5811d6 in call_init (env=0x7fff09e7edc0, argv=0x7fff09e7eda8, argc=2, l=<optimized out>) at dl-init.c:118
#149635 _dl_init (main_map=0x7f32ca59b170, argc=2, argv=0x7fff09e7eda8, env=0x7fff09e7edc0) at dl-init.c:119
#149636 0x00007f32ca57324a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#149637 0x0000000000000002 in ?? ()
#149638 0x00007fff09e7f9db in ?? ()
#149639 0x00007fff09e7f9e3 in ?? ()
#149640 0x0000000000000000 in ?? ()

Trying with CSGO is a somewhat similar backtrace :

#149559 0x00007f98361e0e20 in glXGetProcAddress_renderdoc_hooked (f=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:497
#149560 0x00007f98361e1191 in glXGetProcAddress (f=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:592
#149561 0x00007f98361e0e20 in glXGetProcAddress_renderdoc_hooked (f=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:497
#149562 0x00007f98361e1191 in glXGetProcAddress (f=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:592
#149563 0x00007f98361e0e20 in glXGetProcAddress_renderdoc_hooked (f=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:497
#149564 0x00007f98361e1191 in glXGetProcAddress (f=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:592
#149565 0x00007f98361e0e20 in glXGetProcAddress_renderdoc_hooked (f=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:497
#149566 0x00007f98361e1191 in glXGetProcAddress (f=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:592
#149567 0x00007f98361e0e20 in glXGetProcAddress_renderdoc_hooked (f=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:497
#149568 0x00007f98361e1191 in glXGetProcAddress (f=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:592
#149569 0x00007f98361e0e20 in glXGetProcAddress_renderdoc_hooked (f=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:497
#149570 0x00007f98361e1191 in glXGetProcAddress (f=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:592
#149571 0x00007f98361e1992 in <lambda(char const*)>::operator()(const char *) const (__closure=0x7fff2ed4e510, funcName=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:732
#149572 0x00007f98361e36bd in std::_Function_handler<void*(char const*), GLXHooked(void*)::<lambda(char const*)> >::_M_invoke(const std::_Any_data &, const char *&&) (__functor=..., __args#0=@0x7fff2ed4df00: 0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /usr/include/c++/8/bits/std_function.h:282
#149573 0x00007f98361dbfbb in std::function<void* (char const*)>::operator()(char const*) const (this=0x7fff2ed4e510, __args#0=0x7f9836b81b4c "wglDXSetResourceShareHandleNV") at /usr/include/c++/8/bits/std_function.h:687
#149574 0x00007f98361b1e3a in GLDispatchTable::PopulateWithCallback(std::function<void* (char const*)>) (this=0x7f98370498e0 <GL>, lookupFunc=...) at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/gl_hooks.cpp:179
#149575 0x00007f98361e2f09 in GLXHooked (handle=0x7f9835d42a10) at /home/djdeath/src/mesa-src/renderdoc/renderdoc/driver/gl/glx_hooks.cpp:730
#149576 0x00007f98361dc0d4 in std::_Function_handler<void (void*), void (*)(void*)>::_M_invoke(std::_Any_data const&, void*&&) (__functor=..., __args#0=@0x7fff2ed4e5a0: 0x7f9835d42a10) at /usr/include/c++/8/bits/std_function.h:297
#149577 0x00007f9836b28965 in std::function<void (void*)>::operator()(void*) const (this=0x7fff2ed4e5f0, __args#0=0x7f9835d42a10) at /usr/include/c++/8/bits/std_function.h:687
#149578 0x00007f9836b27c13 in CheckLoadedLibraries () at /home/djdeath/src/mesa-src/renderdoc/renderdoc/os/posix/linux/linux_hook.cpp:119
#149579 0x00007f9836b28109 in LibraryHooks::EndHookRegistration () at /home/djdeath/src/mesa-src/renderdoc/renderdoc/os/posix/linux/linux_hook.cpp:189
#149580 0x00007f9836866d24 in LibraryHooks::RegisterHooks () at /home/djdeath/src/mesa-src/renderdoc/renderdoc/hooks/hooks.cpp:47
#149581 0x00007f9836b3b609 in library_loaded () at /home/djdeath/src/mesa-src/renderdoc/renderdoc/os/posix/posix_libentry.cpp:71
#149582 0x00007f9836b3b763 in init::init (this=0x7f983706dcd0 <do_init>) at /home/djdeath/src/mesa-src/renderdoc/renderdoc/os/posix/posix_libentry.cpp:84
#149583 0x00007f9836b3b678 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at /home/djdeath/src/mesa-src/renderdoc/renderdoc/os/posix/posix_libentry.cpp:85
#149584 0x00007f9836b3b68e in _GLOBAL__sub_I_posix_libentry.cpp(void) () at /home/djdeath/src/mesa-src/renderdoc/renderdoc/os/posix/posix_libentry.cpp:92
#149585 0x00007f98372bb0ca in call_init (l=<optimized out>, argc=argc@entry=3, argv=argv@entry=0x7fff2ed4e828, env=env@entry=0x7fff2ed4e848) at dl-init.c:72
#149586 0x00007f98372bb1d6 in call_init (env=0x7fff2ed4e848, argv=0x7fff2ed4e828, argc=3, l=<optimized out>) at dl-init.c:118
#149587 _dl_init (main_map=0x7f98372d5170, argc=3, argv=0x7fff2ed4e828, env=0x7fff2ed4e848) at dl-init.c:119
#149588 0x00007f98372ad24a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#149589 0x0000000000000003 in ?? ()
#149590 0x00007fff2ed4fa16 in ?? ()
#149591 0x00007fff2ed4fa20 in ?? ()
#149592 0x00007fff2ed4fa74 in ?? ()
#149593 0x0000000000000000 in ?? ()

Repro steps

Just launch one of the 2 games mentioned above with :

renderdoccmd capture /path/to/game

Or from Steam, from the library, right click in the game's name, "set launch options" and write the following :

renderdoc capture %command%

Environment

Renderdoc build from master at : commit 814e1d1b134583aebeebd54167f631bb7d6132e9 (origin/v1.x, origin/HEAD) Author: baldurk baldurk@baldurk.org Date: Thu Oct 25 18:45:20 2018 +0100

Don't pass std::string through StringFormat
Operating System: Debian Linux 64bits
API: OpenGL (with an Intel i965 driver from Mesa3D) at the following revision :

commit cbd44686952b4275d654bcb3555111b412b8c8f4 (origin/master) Author: Jason Ekstrand jason.ekstrand@intel.com Date: Fri Oct 26 13:36:01 2018 -0500

anv: Flag semaphore BOs as external

zao commented 5 years ago

Please note that RenderDoc is intended for capturing your own software as part of your development process.

Capturing third party software is quite outside of the scope of the tool for many reasons, including not knowing what kind of operations and API misuse the programs may be doing, as well as running into anti-hooking mechanisms.

If you're trying to figure out if RenderDoc is working at all on your Linux system, I recommend capturing executables you control that are using core profile OpenGL or Vulkan and has source code available for troubleshooting.

djdeath commented 5 years ago

I would like to argue it's part of my dev process as I'm working on the Mesa 3D i965/anv drivers (both open source projects) and I'm interested to see if renderdoc can be a good tool to reproduce potential driver issues.

I managed to capture from other app, so might be one of the thing you mentioned. Although trying other things like Firefox for instance, seems to disable WebGL support in there.

baldurk commented 5 years ago

As @zao mentions, I do not support or condone capturing copyrighted third party programs for any reason.

I can't say why Firefox disabled WebGL support, it may be that they require some extensions or other support that RenderDoc doesn't handle.

If you find or make a free program that this repros with then I can investigate, although I suspect it may be something systemic which won't be so easy for me to reproduce.

If you want to debug yourself, then you'd need to figure out why the glXGetProcAddress function pointer in the GLX dispatch table ended up pointing to renderdoc's version, as it should point onwards to the 'real' function. That's why this ends up with an infinite recursion. This is how things are expected to work, so if you want to debug you can see where things diverged from this path:

In CheckLoadedLibraries(), we iterate over the list of registered libraries to hook. This is e.g. libGL.so, libGL.so.1, as well as the weird libGLX.so variant. These are all the libraries passed to LibraryHooks::RegisterLibraryHook.
If any of the libraries are already loaded, we first go through all registered function hooks. For each hook, we look it up in the handle obtained in step 1, and store it into the original function pointer if we don't have one yet. So for example glXGetProcAddress is a registered function hook, if it's found in the library handle then we store the function pointer into GLX.glXGetProcAddress. If multiple instances are found the first one 'wins' but we assume they are all equivalently good. So at this point, the library handle should not ever be pointing to librenderdoc.so so GLX.glXGetProcAddress should either end up with NULL if no matching function is found, or point to the real function.
Once all function pointers are filled, we call the registered callback if there is one. This is the call to GLXHooked in the callstacks you pasted.
GLXHooked then uses dlsym (via Process::GetFunctionAddress) to fetch any of the non-hooked functions that we might need in the GLX dispatch table, with GLX_NONHOOKED_SYMBOLS. This is irrelevant for our purposes.
Then we use GLX.glXGetProcAddressARB preferably or else GLX.glXGetProcAddress to fetch any NULL function pointers that we have. This step will fill out any functions like glXCreateContextAttribsARB that may not ever be directly exported by a library.
Finally we iterate over all the GL functions that we want to be able to call, and use the same GLX.glXGetProcAddress to fetch their function pointers.

So somehow in your case, GLX.glXGetProcAddress ended up pointing to renderdoc's version. But it should always be fetched by dlsym on a libGL.so or equivalent handle in step 2. If you figure out why that went wrong,

baldurk commented 5 years ago

Closing this issue due to lack of information to be able to investigate on a supported use-case. If you're able to provide more information feel free to comment here and I can open it up again.

baldurk / renderdoc