baldurk / renderdoc

RenderDoc is a stand-alone graphics debugging tool.
https://renderdoc.org
MIT License
9.03k stars 1.35k forks source link

Crash when an imageless `VkFramebuffer` is passed to `vkBeginCommandBuffer` in `VkCommandBufferInheritanceInfo` #2749

Closed glebov-andrey closed 2 years ago

glebov-andrey commented 2 years ago

Description

The RenderDoc application crashes while loading a capture containing a call to vkBeginRenderPass() for a secondary command buffer when pBeginInfo->pInheritanceInfo->framebuffer is imageless.

The stack trace: (click to expand) ``` renderdoc.dll!VulkanRenderState::SetFramebuffer(WrappedVulkan * vk, ResourceId fb, const VkRenderPassAttachmentBeginInfo *) Line 1001 at c:\build\renderdoc\renderdoc\driver\vulkan\vk_state.cpp(1001) renderdoc.dll!WrappedVulkan::Serialise_vkBeginCommandBuffer(ReadSerialiser & ser, VkCommandBuffer_T *) Line 1181 at c:\build\renderdoc\renderdoc\driver\vulkan\wrappers\vk_cmd_funcs.cpp(1181) renderdoc.dll!WrappedVulkan::ProcessChunk(ReadSerialiser & ser, VulkanChunk chunk) Line 3322 at c:\build\renderdoc\renderdoc\driver\vulkan\vk_core.cpp(3322) [Inline Frame] renderdoc.dll!WrappedVulkan::ContextProcessChunk(ReadSerialiser &) Line 3209 at c:\build\renderdoc\renderdoc\driver\vulkan\vk_core.cpp(3209) renderdoc.dll!WrappedVulkan::ContextReplayLog(CaptureState startEventID, unsigned int endEventID, unsigned int partial, bool) Line 2908 at c:\build\renderdoc\renderdoc\driver\vulkan\vk_core.cpp(2908) renderdoc.dll!WrappedVulkan::ReplayLog(unsigned int startEventID, unsigned int endEventID, ReplayLogType replayType) Line 3909 at c:\build\renderdoc\renderdoc\driver\vulkan\vk_core.cpp(3909) renderdoc.dll!ReplayController::SetFrameEvent(unsigned int eventId, bool force) Line 79 at c:\build\renderdoc\renderdoc\replay\replay_controller.cpp(79) qrenderdoc.exe!CaptureContext::SetEventID::__l2::(IReplayController * r) Line 1609 at c:\build\renderdoc\qrenderdoc\code\capturecontext.cpp(1609) [Inline Frame] qrenderdoc.exe!std::_Func_class::operator()(IReplayController * <_Args_0>) Line 279 at c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional(279) qrenderdoc.exe!ReplayManager::run(int proxyRenderer, const QString & capturefile, const ReplayOptions & opts, std::function progress) Line 497 at c:\build\renderdoc\qrenderdoc\code\replaymanager.cpp(497) [Inline Frame] qrenderdoc.exe!ReplayManager::OpenCapture::__l2::::operator()() Line 58 at c:\build\renderdoc\qrenderdoc\code\replaymanager.cpp(58) [Inline Frame] qrenderdoc.exe!std::_Invoker_functor::_Call(ReplayManager::OpenCapture::__l2:: &) Line 1375 at c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits(1375) [Inline Frame] qrenderdoc.exe!std::invoke(ReplayManager::OpenCapture::__l2:: &) Line 1443 at c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits(1443) [Inline Frame] qrenderdoc.exe!std::_Invoke_ret(std::_Forced) Line 1461 at c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits(1461) qrenderdoc.exe!std::_Func_impl<,std::allocator,void>::_Do_call() Line 214 at c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional(214) [Inline Frame] qrenderdoc.exe!std::_Func_class::operator()() Line 279 at c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional(279) qrenderdoc.exe!LambdaThread::process() Line 511 at c:\build\renderdoc\qrenderdoc\code\qrdutils.h(511) Qt5Core.dll!00007ffaba6d1d49() Qt5Core.dll!00007ffaba504c0f() Qt5Core.dll!00007ffaba505e6f() kernel32.dll!BaseThreadInitThunk() ntdll.dll!RtlUserThreadStart() ```

As far as I can tell, VulkanRenderState::SetFramebuffer assumes that for imageless framebuffers the caller always provides a non-null attachmentsInfo. But WrappedVulkan::Serialise_vkBeginCommandBuffer doesn't do so and ends up defaulting to NULL.

Neither the documentation for VkCommandBufferBeginInfo nor the documentation for VkCommandBufferInheritanceInfo appear to restrict framebuffer to non-imageless framebuffers. The Validation Layers also don't complain about this.

Since neither structure in question appears to have a way of passing VkRenderPassAttachmentBeginInfo (or an equivalent) in their chains, the best option is probably just to avoid filling fbattachments when attachmentsInfo is NULL.

Steps to reproduce

I can't provide a capture of the application, but this is the approximate code to reproduce:

// Create an imageless framebuffer:
vk::RenderPass pass = /* create a render pass with at least one attachment */;
vk::Extent2D extent = /*...*/;
std::vector<vk::FramebufferAttachmentImageInfo> attachment_image_infos = /* collect the image info corresponding to each attachment */;
auto framebuffer = device.createFramebufferUnique(
        vk::StructureChain{
                vk::FramebufferCreateInfo{vk::FramebufferCreateFlagBits::eImageless,
                                          pass,
                                          attachment_count,
                                          nullptr, /*attachments*/
                                          extent.width,
                                          extent.height,
                                          1 /*layers*/},
                vk::FramebufferAttachmentsCreateInfo{attachment_image_infos}}
                .get(),
        nullptr /*allocator*/);

// Allocate and begin a secondary command buffer
auto command_buffer = /* allocate secondary command buffer */;
const auto inheritance_info = vk::CommandBufferInheritanceInfo{pass,
                                                               0, /*subpass*/
                                                               *framebuffer,
                                                               /*...*/};
command_buffer.begin(
        vk::CommandBufferBeginInfo{vk::CommandBufferUsageFlagBits::eOneTimeSubmit | 
                                   vk::CommandBufferUsageFlagBits::eRenderPassContinue,
                                   &inheritance_info});

Passing nullptr instead of framebuffer in VkCommandBufferInheritanceInfo when RenderDoc is loaded into the process works around the issue.

Environment

baldurk commented 2 years ago

Thanks for the clear bug report, that commit fixes all three of the issues you reported. For future reference, there's no need from my side to create separate issues for individual bugs when they are very similar like this, but it's up to you.