Capturing seems to work, but saving doesn't

io7m commented 5 days ago

Description of Bug

I can't seem to get RenderDoc to save captures. I have seen it work a couple of times, but the vast majority of times, it silently fails. I have a very small test application that simply runs for about three frames, at a rate of one frame per second. The application is using vkQueuePresentKHR, so there's no issue delimiting frames. When the application's window appears, the RenderDoc overlay is visible, but pressing F12 or PrtScr to capture a frame does nothing. The setting to automatically capture frames also does nothing. The overlay seems to suggest that it's capturing Vulkan API calls, but no captures are ever saved and so no frames appear in qrenderdoc.

Here's a video of it:

https://ataxia.io7m.com/2024/11/22/renderdoc.mkv

Note the odd flash in the qrenderdoc UI. Normally I would expect to see captured frames appearing in that space, but it's as if the tab opens and then closes again immediately. As mentioned, I have seen it work a couple of times (the 2-3 captured frames appear there), but most of the time it doesn't. Is there some kind of race condition occurring?

I've attached an error report zip below.

Similarly, $ renderdoccmd capture -c /tmp/renderdoc/cap -w --opt-hook-children ./runtest.sh fails to save any captures.

The application is written in Java, calling the Vulkan API through LWJGL. I have to use child process hooking to get the overlay to appear. I believe the VM forks a child process internally.

Steps to reproduce

Unfortunately, I'm not able to share the capture (for obvious reasons), and the application is not in a state where it would be easy for third parties to run as it's still in heavy development. If running the application turns out to be necessary, it'll take me a while to package up something that can be portably run.

Just clicking "Launch" with any combination of settings fails.

I am happy to build explicit support for RenderDoc into the application if there's something I can do to ensure that captures are reliable.

Environment

I have tried the official Linux binaries (https://renderdoc.org/stable/1.35/renderdoc_1.35.tar.gz) and have also tried compiling from the current HEAD, but both versions yield the same problem.

RenderDoc version:

$ renderdoccmd version
renderdoccmd x64 v1.35 built from 8a394019fdb62ad82be87bfd338a5ec43f4a5d95
APIs supported at compile-time: Vulkan, GL, GLES.
Windowing systems supported at compile-time: xlib, XCB, Vulkan KHR_display.

Operating System: Linux
Graphics API: Vulkan

vulkaninfo.txt

renderdoc_report_133457.zip

io7m commented 5 days ago

Attachments were accidentally stripped:

renderdoc_report_133457.zip vulkaninfo.txt

baldurk commented 5 days ago

Some shells will do extremely cursed/terrible things when running programs like exec'ing without fork'ing, which RenderDoc does not support, that might be the problem.

A few things to check:

After running your program, can you connect to it via file -> attach to running instance and trigger a capture that way?
You say that F12/Print screen don't work but if your application renders at an extremely slow pace like one frame per second the keypress may not be detected as it's only checked during present. If you hold down the button to ensure it's seen as pressed during a present does that work?
Does everything work normally if you run a demo program which does not require a shell script wrapper and renders at a normal pace?

If those work and allow you to capture then the problem is limited to making a connection across child processes to your program since you are running it indirectly through a shell script.

Without seeing your shell script it's hard to diagnose exactly, but can you try making sure the shell script does not 'exec' into a program but instead runs it normally. The simplest way would be to include commands before and after (e.g. an echo). Then also run the program explicitly with 'bash' as the executable path and your script as a parameter rather than passing the script - this shouldn't matter but is safer to be sure of what you're testing.

Note you should never use renderdoccmd. This is an undocumented and internal program that is not intended for users to interact with.

io7m commented 5 days ago

Some shells will do extremely cursed/terrible things when running programs like exec'ing without fork'ing, which RenderDoc does not support, that might be the problem.

Thanks, that definitely improves things (although it's not working yet). I was indeed using exec in the shell script as I was under the mistaken impression that this would be easier to trace. :slightly_smiling_face:

Removing exec from the shell script allows the tab to appear in qrenderdoc for at least as long as the program runs. I'll try your other suggestions and see if I can get a saved capture out of it.

Note you should never use renderdoccmd. This is an undocumented and internal program that is not intended for users to interact with.

Noted!

io7m commented 5 days ago

It does indeed seem to be the application's frame rate that's causing this. If I, for example, add a little loop to spin for ~180 frames at 60fps, I can trigger captures within those frames using the keyboard.

I'm essentially tracing unit tests, so each test is only 1-3 frames in total. I suspect I really need to be using the RenderDoc API for what I'm trying to do.

Thanks!

baldurk / renderdoc