Cannot initialize RenderDoc in-app API with GLFW application on PopOS 22.04

[KitsuneAlex]

Description

I am developing an OpenGL application on Linux using the Kotlin programming language and the LWJGL bindings for GLFW and OpenGL.

I've ran into an issue where i cannot successfully use the RenderDoc in-app API to attach to my application. After consulting the RenderDoc API docs, i've used SetActiveWindow to explicitly set the active GLX window/drawable and context, however this also seemed without any effect.

Instead of relying on the outdated RenderDoc versions provided by my distribution vendor, i already tried the latest stable binaries (1.35) from the homepage and i've also built RenderDoc from source sepcifically for my system, since i was fearing some issue due to outdated X11 binaries.

I am very close to giving up now so i am consulting the issue section in the hope that this is some type of bug, either due to my distribution or something else involving the GL context hook process.

It is noteworthy that the same code i wrote works fine on Windows.

Steps to reproduce

• Start my JVM application using Java 17
• Before any calls to GLFW or GL are made, the RenderDoc API is initialized on the main thread (RENDERDOC_GetAPI) - i used JNA to model the RenderDoc API which in turn uses JNI to load the librenderdoc.so shared object into the JVM process
• GLFW window is created, GL 4.6 Core w/ FWC
• GL capabilities are initialized
• SetActiveWindow is called with a valid GLX context as the device and the GLX window/drawable as the window
• QRenderDoc can connect to the process but reports API: None, no overlay is visible in the application

Environment

• RenderDoc version: 1.35 built from 647313ae
• Operating System: PopOS 22.04
• Graphics API: OpenGL 4.6, Core Profile, Forward Compatibility
• GPU: NVIDIA GTX Titan X 12GB
• Driver: NVIDIA 555.58.02/NVML 12.555.58.02/NVC 1.29
• X Server: 1.21.1.4 (12101004)

[baldurk]

You've omitted the most crucial part of your steps to reproduce which is including your application. Without that it's impossible to follow your steps meaningfully.

I'm also not clear from your description exactly where the problem is - you mostly talk about issues with using the in-application API but I'm not sure if that's the only problem since you say that the UI doesn't display an API. Can you start from the basics - remove all uses of the in-application API and just launch your program from the UI. Does that break? Using the API is an optional extra step and can't break capturing so it sounds like something else is going on.

[KitsuneAlex]

Yes, something else is definitly going on outside the API and my code. It seems to be a general issue on my system where i'm not able to capture any GL applications, even if launched from the RenderDoc UI. That is, applications built with GLFW. SDL2 seems to have no issues weirdly enough.

I will include an application to reproduce later since i am still working right now, thanks for your quick response! :)

[demensdeum]

@KitsuneAlex I had mixing Vulkan and OpenGL initialization, which leads to no capture in renderdoc.

[KitsuneAlex]

@KitsuneAlex I had mixing Vulkan and OpenGL initialization, which leads to no capture in renderdoc.

How would i identify such issue?

[demensdeum]

@KitsuneAlex I had to check my initialization code, because for some reason there was no problems with basic OpenGL rendering over Vulkan context, but only with graphics debugging tools. I just initialized SDL_Window with VULKAN flag + vulkan instances and extensions from old code, and initialized OpenGL context on top of that; so I had to remove Vulkan initialization and change window flag to OpenGL. If your Kotlin app environment do mix of Vulkan initialization and OpenGL rendering, then you possibly get into same situation as me. Here is my changes to fix that: https://github.com/demensdeum/SurrealEngine-Emscripten/commit/32c09781b547f0a7de809918b2ee4cc27f2cd32b