Editor crashes very often in Forward+ and Mobile due to FPS Monitor Vulkan overlay

[Exorion1er]

Godot version

4.2.dev.custom_build [b467afe65] (no edits)

System information

Godot v4.2.dev (b467afe65) - Windows 10.0.19044 - Vulkan (Forward+) - dedicated NVIDIA GeForce GTX 1070 (NVIDIA; 31.0.15.3734) - Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz (8 Threads)

Issue description

I'm trying out Godot for the first time, so I went with Godot 4 Latest, and it just keeps on crashing again and again at different times doing different things. It does that on both Mobile and Forward+, but it doesn't on compatibility. It happens infrequently. I do need to click something or do something for it to crash, meaning that if I leave the software open without doing anything it won't crash by itself, it needs input, but that input has come on multiple different UIs and things, I can't pinpoint one exactly. I could just reopen it and keep going but I'd need to do that very, very often. Like, I can't do anything productive, it crashes way too often.

What I mean by crash: The editor just fully closes, no error or anything. It does freeze, but for less than a second then just closes.

It happens on both brand new and populated projects. I've tried using the console, nothing comes up. Same thing with --verbose.

But, with the help of QbieShay on Discord, we've figured out by building the source and following the stacktrace that it was a Vulkan issue. From there, I enabled Vulkan Validation (https://docs.godotengine.org/en/stable/contributing/development/debugging/vulkan/vulkan_validation_layers.html). After that, it stopped crashing but instead gave me errors in the console output: message.txt

Also with the help of QbieShay, building the source with debug symbols allowed us to figure out that the error is thrown from this line : https://github.com/godotengine/godot/blob/787259441abb7aa92a382ccf48591a70136f25f0/drivers/vulkan/vulkan_context.cpp#L1788

If you need any more information or logs/outputs, don't hesitate.

Steps to reproduce

There are many ways to make it crash, it happens during a lot of interaction with the UI, but here's one I found that is consistent:

• Open or Create a project (happens consistently on both cases)
• Select a node in the scene
• In the Inspector, click the Add Metadata button
• Choose any element in the Type listbox The app crashes at this point

Minimal reproduction project

N/A

[akien-mga]

Thanks for the detailed bug report! CC @godotengine/rendering

Did you check if the issue is reproducible in the current stable version of Godot? 4.1.1-stable at this time: https://godotengine.org/download/

[Exorion1er]

It is, in both 4.1.1, 4.1 and 4.0 I'm afraid. I've also tried with the steam edition (no idea if it's any different), and the same thing happens.

[Exorion1er]

Edit: Added line that throws the first error in the original post Edit2: Added a consistent way to reproduce the bug

I'm assuming this is something related to my hardware in some way, so if you need any help to try a potential fix or anything, don't hesitate, I'm somewhat technical, just not with Vulkan or C++.

[Calinou]

Do you have any third-party Vulkan layers installed, such as RTSS, Epic Games Launcher/Rockstar Games Launcher, or software that injects itself into processes in general?

I think Vulkan Configurator (which is part of the Vulkan SDK) can display a list of those.

I've also tried with the steam edition (no idea if it's any different)

Official Godot binaries are exactly the same on Steam and non-Steam.

[Exorion1er]

I do have 2 of those installed, both Epic Games Launcher and Rockstar Game Launcher just not open at the moment. Do I need to uninstall them ?

Vulkan Configurator Log:

Vulkan Development Status:
- Layers override: "Validation" configuration
- VULKAN_SDK environment variable: C:\VulkanSDK\1.3.261.1
- Vulkan Loader version: 1.3.241
- User-Defined Layers locations:
    - VK_LAYER_PATH variable: None
    - Per-configuration paths: None
    - VK_ADD_LAYER_PATH variable: None
- `vk_layer_settings.txt` uses the default platform path:
    C:\Users\<username>\AppData\Local\LunarG\vkconfig\override
- Available Layers:
    - VK_LAYER_NV_optimus
    - VK_LAYER_fpsmon
    - VK_LAYER_OBS_HOOK
    - VK_LAYER_EOS_Overlay
    - VK_LAYER_VALVE_steam_overlay
    - VK_LAYER_VALVE_steam_fossilize
    - VK_LAYER_EOS_Overlay
    - VK_LAYER_EOS_Overlay
    - VK_LAYER_ROCKSTAR_GAMES_social_club
    - VK_LAYER_LUNARG_api_dump
    - VK_LAYER_LUNARG_gfxreconstruct
    - VK_LAYER_KHRONOS_synchronization2
    - VK_LAYER_KHRONOS_validation
    - VK_LAYER_LUNARG_monitor
    - VK_LAYER_LUNARG_screenshot
    - VK_LAYER_KHRONOS_profiles
    - VK_LAYER_KHRONOS_shader_object
- Physical Devices:
    - NVIDIA GeForce GTX 1070 with Vulkan 1.3.242
        - deviceUUID: 356852A47AF2B7E2AA2EBB26B0CBAAAE
        - driverUUID: FC1BA415F95957669DE6E170FEA5D8D3

[Calinou]

I'd try seeing if there's a way to disable all third-party layers without removing the software that installed them, including OBS and fpsmon which I notice in the log.

IIRC, you can use environment variables for this.

[Exorion1er]

That worked ! I ran set VK_DEVICE_LAYERS=VK_LAYER_KHRONOS_validation Godot_v4.1.1-stable_win64.exe before running Godot and the crashes just stopped now !

So does that mean it's one of those layers that's incompatible with Godot then ? I don't use fpsmon so I can remove that but the others from what I can figure out online are stuff that I use every now and then

[Exorion1er]

Ok after some trial and error, it looks like the culprit was indeed FPS Monitor. I've uninstalled it, no error anymore. If I install the latest version of it, no error either, BUT if I install the version I previously had installed, then Godot crashes. This means, I'm guessing, that that specific FPS Monitor version, even when closed, broke Godot.

Never in a million years would I have guessed that 😆 Anyway, guess that's fixed, thanks a lot 😅

[bitsawer]

Not the first time third-party layers have caused issues. I wonder if we could somehow try to log or detect and warn users if they have certain (or any uncommon) Vulkan layers in use when starting Godot.

[clayjohn]

Maybe @darksylinc knows?

[darksylinc]

Infrequent crashes on vkDestroyRenderPass is definitely a symptom of the sync bug fixed by #80566 (see that #80550 mentions sync problems in vkDestroyRenderPass).

I do need to click something or do something for it to crash, meaning that if I leave the software open without doing anything it won't crash by itself, it needs input, but that input has come on multiple different UIs and things, I can't pinpoint one exactly

The most likely reason behind this is that by default Godot only updates on user input. If you were to change Project Settings to "Update Continuously", you'll probably see this crash after leaving the Editor idle (without doing anything) for a couple minutes.

You can try compiling the branch https://github.com/darksylinc/godot/tree/matias-vulkan-sync-fix to see if the problem goes away permanently (knocks on wood).

I was planning on rebasing that branch tomorrow with latest master, incorporate the suggested edits; and strongly suggest @clayjohn to merge it ASAP, considering the current situation where, due to a sad reason, a large influx of users are flooding to Godot for the first time.

Having that said, I'm a little worried about this bit:

Validation Error: [ VUID-vkDestroySemaphore-semaphore-parameter ] Object 0: handle = 0x28f7564ada0, type = VK_OBJECT_TYPE_INSTANCE; | MessageID = 0x3f4efd34 | vkDestroySemaphore(): Invalid VkSemaphore Object 0x5987ab0000004696. The Vulkan spec states: If semaphore is not VK_NULL_HANDLE, semaphore must be a valid VkSemaphore handle (https://vulkan.lunarg.com/doc/view/1.3.261.1/windows/1.3-extensions/vkspec.html#VUID-vkDestroySemaphore-semaphore-parameter)

In all of Godot we call vkDestroySemaphore on shutdown (doesn't apply here) and when a window is destroyed (which happens often because every pop up windows has its own window).

I can see that destruction is the following:

for (uint32_t i = 0; i < frame_count; i++) {
    // Destroy the semaphores now (we'll re-create it later if we have to).
    // We must do this because the semaphore cannot be reused if it's in a signaled state
    // (which happens if vkAcquireNextImageKHR returned VK_ERROR_OUT_OF_DATE_KHR or VK_SUBOPTIMAL_KHR)
    // The only way to reset it would be to present the swapchain... the one we just destroyed.
    // And the API has no way to "unsignal" the semaphore.
    vkDestroySemaphore(device, window->image_acquired_semaphores[i], nullptr);
    window->image_acquired_semaphores[i] = 0;
}

This rules out double frees because we immediately set window->image_acquired_semaphores to VK_NULL_HANDLE (0).

However, I can theorize that if a window is destroyed before image_acquired_semaphores is populated:

for (uint32_t i = 0; i < frame_count; i++) {
    VkResult vkerr = vkCreateSemaphore(device, &semaphoreCreateInfo, nullptr, &window->image_acquired_semaphores[i]);
    ERR_FAIL_COND_V(vkerr, ERR_CANT_CREATE);
}

Then the image_acquired_semaphores will be left uninitialized and we will later try to destroy it. Error checks a little earlier suggests I'm spot on, like:

if (window->width == 0 || window->height == 0) {
    free(presentModes);
    // Likely window minimized, no swapchain created.
    return ERR_SKIP;
}

Which are a red flag that this is happening so we definitely need to zero-initialize image_acquired_semaphores. I will fix that tomorrow.

Another possibility is that window creation & destruction are being called from different threads which should definitely not happen. I should check if the thread safe macros are placed.

As a workaround you can launch Godot with --single-window which will minimize this type of bug.