[capture] Can't capture vkcube and resize the window when VK_LAYER_OBS_HOOK is active

[bradgrantham-lunarg]

Describe the capture bug:

$ gfxrecon.py capture vkcube Warning: the 'capture' command is deprecated and will be removed in a future update. Use 'capture-vulkan' instead. Executing program C:\VulkanSDK\1.3.246.0\Bin\vkcube.EXE Errors: Selected GPU 0: NVIDIA GeForce RTX 2070 SUPER, type: DiscreteGpu

Output: [gfxrecon] INFO - Settings Loader: Found option "GFXRECON_CAPTURE_FILE" with value "F:\gfxreconstruct\gfxrecon_capture.gfxr" [gfxrecon] INFO - Initializing GFXReconstruct capture layer [gfxrecon] INFO - GFXReconstruct Version 0.9.20-dev (test-SDK-integration:3a665ea) [gfxrecon] INFO - Recording graphics API capture to F:\gfxreconstruct\gfxrecon_capture_20230628T110504.gfxr [gfxrecon] INFO - Vulkan Header Version 1.3.246 [gfxrecon] WARNING - Create a duplicated Handle: 1766733260560. This wrapper can't be written into VulkanStateHandleTable. [gfxrecon] WARNING - Create a duplicated Handle: 1766791439360. This wrapper can't be written into VulkanStateHandleTable. [gfxrecon] WARNING - GetWrappedId() couldn't find Handle: 1766733260560's wrapper. It might have been destroyed [gfxrecon] WARNING - GetWrappedId() couldn't find Handle: 1766791439360's wrapper. It might have been destroyed [gfxrecon] WARNING - GetWrappedId() couldn't find Handle: 1766733260560's wrapper. It might have been destroyed [gfxrecon] WARNING - GetWrappedId() couldn't find Handle: 1766791439360's wrapper. It might have been destroyed

Build Environment: SDK 1.3.246.0

Capture Settings: none

To Reproduce: Capture cube and then resize its window

[davidlunarg]

I tried this, but didn't see any problems:

C:\Users\David\TMP>gfxrecon.py capture vkcube Warning: the 'capture' command is deprecated and will be removed in a future update. Use 'capture-vulkan' instead. Executing program C:\VulkanSDK\1.3.246.0\Bin\vkcube.EXE Output: [gfxrecon] INFO - Settings Loader: Found option "GFXRECON_CAPTURE_FILE" with value "C:\Users\David\TMP\gfxrecon_capture.gfxr" [gfxrecon] INFO - Initializing GFXReconstruct capture layer [gfxrecon] INFO - GFXReconstruct Version 0.9.20-dev (test-SDK-integration:81f9f8e) [gfxrecon] INFO - Recording graphics API capture to C:\Users\David\TMP\gfxrecon_capture_20230710T131619.gfxr [gfxrecon] INFO - Vulkan Header Version 1.3.246

C:\Users\David\TMP>dir Volume in drive C has no label. Volume Serial Number is 30B0-DC74

Directory of C:\Users\David\TMP

07/10/2023 01:16 PM

. 07/10/2023 01:16 PM .. 07/10/2023 01:16 PM 15,557,879 gfxrecon_capture_20230710T131619.gfxr 1 File(s) 15,557,879 bytes 2 Dir(s) 359,241,863,168 bytes free

C:\Users\David\TMP>gfxrecon replay gfxrecon_capture_20230710T131619.gfxr Total time: 16.094511 seconds Replay FPS: 58.342872 fps, 16.094511 seconds, 939 frames, framerange 1-939

I resized the window several times while creating the capture. When I replayed the capture file, the window was resized in the same way as when I created the capture. The original report has some warnings generated during capture, which I did not get. Maybe I'm not doing exactly what Brad did?

[bradgrantham-lunarg]

Thanks, @davidlunarg ! On Wednesday I can try with dev and make sure I wasn't running with something weird in my tree.

[bradgrantham-lunarg]

dev 06e7639 is still behaving this way for me, crashing in:

    VkLayer_gfxreconstruct.dll!std::vector<std::vector<unsigned char,std::allocator<unsigned char>>,std::allocator<std::vector<unsigned char,std::allocator<unsigned char>>>>::_Tidy() Line 1763    C++
    VkLayer_gfxreconstruct.dll!std::vector<std::vector<unsigned char,std::allocator<unsigned char>>,std::allocator<std::vector<unsigned char,std::allocator<unsigned char>>>>::~vector<std::vector<unsigned char,std::allocator<unsigned char>>,std::allocator<std::vector<unsigned char,std::allocator<unsigned char>>>>() Line 714    C++
    VkLayer_gfxreconstruct.dll!gfxrecon::encode::HandleUnwrapMemory::~HandleUnwrapMemory()  C++
    VkLayer_gfxreconstruct.dll!gfxrecon::encode::ImageWrapper::~ImageWrapper()  C++
    VkLayer_gfxreconstruct.dll!gfxrecon::encode::ImageWrapper::`scalar deleting destructor'(unsigned int)   C++
>   VkLayer_gfxreconstruct.dll!gfxrecon::encode::DestroyWrappedHandle<gfxrecon::encode::SwapchainKHRWrapper>(VkSwapchainKHR_T * handle) Line 614    C++
    VkLayer_gfxreconstruct.dll!gfxrecon::encode::DestroySwapchainKHR(VkDevice_T * device, VkSwapchainKHR_T * swapchain, const VkAllocationCallbacks * pAllocator) Line 8042 C++

Before this crash, I see messages of the type:

[gfxrecon] WARNING - Create a duplicated Handle: 1960993440688. This wrapper can't be written into VulkanStateHandleTable.

[bradgrantham-lunarg]

Only happens when VK_LAYER_OBS_HOOK is enabled.

When not enabled, I can resize the window as much as desired. I also do not see any messages about "duplicated Handle".

When VK_LAYER_OBS_HOOK is enabled, after the swapchain is created and destroyed a few times I see our loop on image_wrapper in DestroyWrappedHandle<SwapchainKHRWrapper> appears to somehow have a handle in it to a VkSwapchain.

Further work would include: ~* make sure OBS hasn't fixed this bug by upgrading OBS~ (29.1 also shows this problem)

• see if we can identify why wrapper->child_images appears to have an entry that is either a VkSwapchainWrapper or is a VkImageWrapper that contains the handle to a VkSwapchain.

vkcube isn't multithreaded by itself, but I suppose it's possible that OBS hook starts a separate thread and therefore causes some kind of problem related to https://github.com/LunarG/gfxreconstruct/issues/1189.

[bradgrantham-lunarg]

Updated OBS to 29.1, still happens.

FYI one can disable their layer with VK_LOADER_LAYERS_DISABLE=VK_LAYER_OBS_HOOK.

[davidlunarg]

Here's the sequence that occurs in the capture layer when the OBS hook layer is enabled:

1. vkCreateSwapchainKHR  - returns S1
2. vkGetSwapchainImagesKHR(S1)
         returns 3 images: I1,I2,I3
3. vkCreateSwapchainKHR  - returns S2
        pCreateInfo->oldSwapchain is set to S1 
4. vkGetSwapchainImagesKHR(S2)
         returns 3 images: I1,I4,I5
5. vkDestroySwapchain(S1) 

The problem is that vkGetSwapchainImagesKHR returns I1 for both S1 and S2. This results in a warning being generated by the capture layer during vkGetshainImagesKHR(S2) capture because it encountered a duplicate handle. The image handle is not saved with S2, and later, when swapchain S2 is destroyed, the capture layer crashes while trying to access the 3 images it thinks it saved for S2.

I also observed that if an image is returned by vkGetAcquireNextImage for a swapchain, that image can still be returned by vkGetSwapchainImages for a different swapchain

The problem seems to happen only on nVidia, not AMD.

The capture layer needs to be modified to handle the case when an image is associated with two different swapchains.

[davidlunarg]

Attached is a modified cube.c that shows the problem without having to enable the obs hook layer. cube.c.txt

[ziga-lunarg]

Should be fixed with #1424