[Libretro] Crash when using AI translate with vulkan driver

[Sanaki]

Platform / OS / Hardware: x86_64 Linux (Garuda/Arch)

Github hash: 44fa364f36c43bed19b055096600f075c656f78c

Hardware: GTX 1060 6GB

Description of the Issue

When attempting to translate foreign games using RetroArch's AI service with the vulkan driver, the core crashes. A third party reported the same behavior on Windows. Screenshots don't cause the same crash.

Debugging Steps Tested

• Set up RetroArch AI service and bind a key to translate
• Start any game in the Flycast core
• Press the translation key
• If it doesn't crash (rarely it will succeed), repeat as necessary

Logs Gathered

GDB backtrace was unsuccessful, core behaves differently when being debugged (crash on fast forward, stall without full abort on AI translate attempt, etc)

Non-debugged crash lines (different runs):

munmap_chunk(): invalid pointer
free(): invalid size
free(): invalid pointer

Vulkan debug output on AI translate attempt:

[ERROR] [Vulkan]: Validation Error: Validation Error: [ VUID-vkResetCommandBuffer-commandBuffer-00045 ] Object 0: handle = 0x55555a5fd3b0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x1a8e390000000a56, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0x1e7883ea | vkResetCommandBuffer():  (VkCommandBuffer 0x55555a5fd3b0[]) is in use. The Vulkan spec states: commandBuffer must not be in the pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkResetCommandBuffer-commandBuffer-00045)
[ERROR] [Vulkan]: Validation Error: Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00049 ] Object 0: handle = 0x55555a5fd3b0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID =0x84029a9f | vkBeginCommandBuffer():  on active VkCommandBuffer 0x55555a5fd3b0[] before it has completed. You must check command buffer fence before this call. The Vulkan spec states: commandBuffer must not be in the recording or pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00049)
[ERROR] [Vulkan]: Validation Error: Validation Error: [ VUID-vkQueueSubmit-fence-00064 ] Object 0: handle = 0x555558d1d780, type = VK_OBJECT_TYPE_QUEUE; Object 1: handle = 0xdee72200000009e1, type = VK_OBJECT_TYPE_FENCE; | MessageID = 0xbf76c98d | vkQueueSubmit():  (VkFence 0xdee72200000009e1[]) is already in use by another submission. The Vulkan spec states: If fence is not VK_NULL_HANDLE, fence must not be associated with any other queue command that has not yet completed execution on that queue (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkQueueSubmit-fence-00064)
[ERROR] [Vulkan]: Validation Error: Validation Error: [ VUID-vkQueueSubmit-pCommandBuffers-00071 ] | MessageID = 0x2e2f4d65 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] VkCommandBuffer 0x55555a5fd3b0[] is already in use and is not marked for simultaneous use. The Vulkan spec states: If any element of the pCommandBuffers member of any element of pSubmits was not recorded with the VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT, it must not be in the pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkQueueSubmit-pCommandBuffers-00071)
[ERROR] [Vulkan]: Validation Error: Validation Error: [ VUID-vkQueuePresentKHR-pWaitSemaphores-03268 ] Object 0: handle = 0x21c48d00000009de, type = VK_OBJECT_TYPE_SEMAPHORE; Object 1: handle = 0x555558d1d780, type = VK_OBJECT_TYPE_QUEUE; | MessageID = 0x251f8f7a | vkQueuePresentKHR(): pPresentInfo->pWaitSemaphores[0] queue (VkQueue 0x555558d1d780[]) is waiting on semaphore (VkSemaphore 0x21c48d00000009de[]) that has no way to be signaled. The Vulkan spec states: All elements of the pWaitSemaphores member of pPresentInfo must reference a semaphore signal operationthat has been submitted for execution and any semaphore signal operations on which it depends must have also been submitted for execution (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkQueuePresentKHR-pWaitSemaphores-03268)

[bslenul]

On Windows: crashes with Vulkan too, with D3D11 no crash but you get a "Could not read viewport" error notification, with GLCore however it seems to work, although the translation is non-sensical since the translation server receives an upside-down image to translate :p

[PCdasala]

just to complement, when I use dx11 or another dx the message is "clould not read viewport", with vulcan a meaningless translation attempt occurs and the game continues to bug and close. This only happens on dreamcast and ps2

[flyinghead]

Very likely a RetroArch issue since it happens with several cores.

[bslenul]

Yeah the D3D11 error is a RetroArch thing, you can't even take a regular screenshot with it for whatever reason with any HW rendered core, no screenshot = no translation.

So if you really need translation, atm the only solution seems to be downgrading to RetroArch 1.16.0 (because of the upside-down issue) and use the GL renderer (to avoid crashes and viewport errors).

[bslenul]

About the Vulkan crash, in my case it seems tied to the "Delay Frame Swapping" core option, if I disable it it doesn't crash anymore.

[bslenul]

While taking a screenshot for the RetroArch upside-down translation issue (https://github.com/libretro/RetroArch/issues/16275) I noticed that it didn't crash with an older RA version, it looks like the Vulkan crash is caused by the same commit and isn't a Flycast issue. At least on Windows 10, haven't tested any other OS.

[Sanaki]

I'll trust your guidance on this one. Closing.