baldurk / renderdoc

RenderDoc is a stand-alone graphics debugging tool.
https://renderdoc.org
MIT License
9.02k stars 1.35k forks source link

replay attempts to create mutable-format imageview from non-mutable image #3094

Closed zmike closed 1 year ago

zmike commented 1 year ago

Description

Trying to click on the texture viewer and renderdoc crashes due to spec violation

Steps to reproduce

#0  0x00007ffff16b0884 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007ffff165fafe in raise () from /lib64/libc.so.6
#2  0x00007ffff164887f in abort () from /lib64/libc.so.6
#3  0x00007ffff164879b in __assert_fail_base.cold () from /lib64/libc.so.6
#4  0x00007ffff1658187 in __assert_fail () from /lib64/libc.so.6
#5  0x00007fffa90b74a6 in vk_image_view_init (device=0x7fff74f2e720, image_view=0x7fff2cafcd00, driver_internal=false, pCreateInfo=0x7fff74ba5640)
    at ../src/vulkan/runtime/vk_image.c:451
#6  0x00007fffa8fbc258 in radv_image_view_init (iview=0x7fff2cafcd00, device=0x7fff74f2e720, pCreateInfo=0x7fff74ba5640, img_create_flags=0, 
    extra_create_info=0x7fff837fd064) at ../src/amd/vulkan/radv_image.c:2133
#7  0x00007fffa8fbd844 in radv_CreateImageView (_device=0x7fff74f2e720, pCreateInfo=0x7fff74ba5640, pAllocator=0x0, pView=0x7fff2ca12b00)
    at ../src/amd/vulkan/radv_image.c:2577
#8  0x00007ffff3fc3b12 in WrappedVulkan::vkCreateImageView(VkDevice_T*, VkImageViewCreateInfo const*, VkAllocationCallbacks const*, VkImageView_T**) ()
   from /usr/local/lib64/librenderdoc.so
#9  0x00007ffff3b7f0ad in VulkanReplay::CreateTexImageView(VkImage_T*, VulkanCreationInfo::Image const&, CompType, VulkanReplay::TextureDisplayViews&) ()
   from /usr/local/lib64/librenderdoc.so
#10 0x00007ffff3b7f493 in VulkanReplay::RenderTextureInternal(TextureDisplay, ImageState const&, VkRenderPassBeginInfo, int) ()
   from /usr/local/lib64/librenderdoc.so
#11 0x00007ffff3b81dcd in VulkanReplay::RenderTexture(TextureDisplay) () from /usr/local/lib64/librenderdoc.so
#12 0x00007ffff45576a4 in ReplayOutput::DisplayContext() () from /usr/local/lib64/librenderdoc.so
#13 0x0000000000c8606c in TextureViewer::RT_PickPixelsAndUpdate(IReplayController*) ()
#14 0x0000000000b515b6 in ReplayManager::run(int, QString const&, ReplayOptions const&, std::function<void (float)>) ()
#15 0x0000000000b51e75 in std::_Function_handler<void (), ReplayManager::OpenCapture(QString const&, ReplayOptions const&, std::function<void (float)>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
#16 0x0000000000b5370d in LambdaThread::process() ()
#17 0x00007ffff1ee8481 in void doActivate<false>(QObject*, int, void**) () from /lib64/libQt5Core.so.5
#18 0x00007ffff1cf35bd in QThread::started(QThread::QPrivateSignal) () from /lib64/libQt5Core.so.5
#19 0x00007ffff1cf59c6 in QThreadPrivate::start(void*) () from /lib64/libQt5Core.so.5
#20 0x00007ffff16ae947 in start_thread () from /lib64/libc.so.6
#21 0x00007ffff1734870 in clone3 () from /lib64/libc.so.6
(gdb) fr 5
#5  0x00007fffa90b74a6 in vk_image_view_init (device=0x7fff74f2e720, image_view=0x7fff2cafcd00, driver_internal=false, pCreateInfo=0x7fff74ba5640)
    at ../src/vulkan/runtime/vk_image.c:451
451          assert(image_view->format == image->format);
(gdb) p image_view->format
$1 = VK_FORMAT_R16G16B16A16_UNORM
(gdb) p image->format
$2 = VK_FORMAT_R16G16B16A16_SFLOAT

Environment

3ca187d725d9cfc27145198106e70ce2e27701a9

baldurk commented 1 year ago

That commit should fix the issue, it was inheriting the display typecast from the underlying texture and incorrectly using that when displaying the overlay texture.