Vulkan App enabling renderdoc layer causes ERROR_INITIALIZATION_FAILED

[Firestar99]

Description

Creating a vulkan instance with VK_LAYER_RENDERDOC_Capture specified as a layer causes it to error with ERROR_INITIALIZATION_FAILED, with no further detail provided.

However, every other way to start an application with renderdoc functions correctly:

• launching through Renderdoc's UI
• vkconfig with VK_LAYER_RENDERDOC_Capture force enabled
• LD_PRELOAD=librenderdoc.so as env var
• ENABLE_VULKAN_RENDERDOC_CAPTURE=1 as env var

If the vulkan application specifies the renderdoc layer, it will always error, regardless if any of the other methods listed above would have enabled renderdoc anyways.

This only started happening since I updated my Renderdoc to 1.32 today, previously I ran a 1.30 beta build with commit 3eccdc736ebe4a7a943127bbea279ce2d7b39732.

Steps to reproduce

Based on SaschaWillems' Vulkan Examples

1. In base/vulkanexamplebase.cpp l.131 change the string VK_LAYER_KHRONOS_validation to VK_LAYER_RENDERDOC_Capture
2. Choose any example, like examples/triangle
3. In it's VulkanExample constructor (l.116 in the triangle example) add settings.validation = true;
4. run the example and observe in stderr:

   
   ERROR: [1][

Could not create Vulkan instance : ERROR_INITIALIZATION_FAILED

<!-- Please list the steps that someone can take to reproduce the bug. -->

<!-- If you can share your capture or your application, PLEASE DO THAT NOW. It is by far the easiest way to demonstrate a bug. You can share it privately via email to baldurk@baldurk.org and mention it here. -->
<!-- If you cannot share because of privacy or other reasons, please state that and give as much extra information as you can. -->
<!-- Steps like "run my application" or "load this capture" are not useful unless you share the application or capture. Be specific! -->

## Environment

<!-- if you are running a nightly build, list the date or commit hash for the version -->

* RenderDoc version: 1.32
* Operating System: Ubuntu 32.10, KDE Plasma via wayland
* Graphics API: Vulkan, [Mesa RADV 23.2.1-1ubuntu3.1](http://vulkan.gpuinfo.org/displayreport.php?id=25819), [AMDVLK 2024.Q1.2](http://vulkan.gpuinfo.org/displayreport.php?id=30195)
* Vulkan SDK: 1.3.280.0

<!-- More details here never hurt! For example your GPU, driver version, etc. -->

[baldurk]

Only the first method you mentioned is supported and intended to work. None of the others are documented and there is nothing anywhere to suggest these should work. Trying to hack RenderDoc to work in unsupported ways may misbehave or crash. Generally I do not recommend ever having vkconfig open at the same time as RenderDoc as it has caused a number of problems in the past.

If you want to capture your program from RenderDoc you should launch it from the UI. Never specify the layer manually or try to enable internal undocumented functionality.

[Firestar99]

If the only official way to launch an App with RenderDoc is via it's UI, there does not seem to be a way an application can programmatically attach RenderDoc, is there?

[baldurk]

You can save a capture settings file for your application with 'auto start' enabled and launch the UI program with that settings file, which will then run your program.

Otherwise you can use the python replay API, which has access to do capturing, replaying and analysis all programmatically which is how the automated tests run.