godotengine / godot

Godot Engine – Multi-platform 2D and 3D game engine
https://godotengine.org
MIT License
90.46k stars 21.07k forks source link

Editor crash when updating NVIDIA Drivers: GPU process exited unexpectedly #98386

Open allenwp opened 1 day ago

allenwp commented 1 day ago

Tested versions

NVIDIA Driver versions:

System information

Windows 10.0.22631 - Multi-window, 2 monitors - Vulkan (Forward+) - dedicated NVIDIA GeForce GTX 980 Ti (NVIDIA; 32.0.15.6094) - 13th Gen Intel(R) Core(TM) i7-13700K (24 threads)

Issue description

I was working in VS Code on GDScript changes and suddenly both of my monitors went black. It seems that my NVIDIA graphics drivers automatically updated in the background.

Here is the crash log from Godot:

graphics_device_crash_godot_log-44fa55234.txt

And here's a highlight of the last messages:

...
   at: RenderingDevice::draw_list_bind_uniform_set (servers\rendering\rendering_device.cpp:4176)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_render_pipeline (servers\rendering\rendering_device.cpp:4081)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_set_push_constant (servers\rendering\rendering_device.cpp:4291)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_vertex_array (servers\rendering\rendering_device.cpp:4214)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_index_array (servers\rendering\rendering_device.cpp:4247)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_draw (servers\rendering\rendering_device.cpp:4313)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_uniform_set (servers\rendering\rendering_device.cpp:4176)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_render_pipeline (servers\rendering\rendering_device.cpp:4081)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_set_push_constant (servers\rendering\rendering_device.cpp:4291)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_index_array (servers\rendering\rendering_device.cpp:4247)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_draw (servers\rendering\rendering_device.cpp:4313)
ERROR: Immediate draw list is already inactive.
   at: (servers\rendering\rendering_device.cpp:4526)
ERROR: Last known breadcrumb: BLIT_PASS
   at: RenderingDeviceDriverVulkan::print_lost_device_info (drivers\vulkan\rendering_device_driver_vulkan.cpp:5121)
ERROR: VK_EXT_device_fault not available.
   at: RenderingDeviceDriverVulkan::on_device_lost (drivers\vulkan\rendering_device_driver_vulkan.cpp:4987)
ERROR: Vulkan device was lost.
   at: (drivers\vulkan\rendering_device_driver_vulkan.cpp:2457)

================================================================
CrashHandlerException: Program crashed
Engine version: Godot Engine v4.4.dev.custom_build (44fa552343722bb048e2d7c6d3661174a95a8a3c)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[0] RenderingDeviceDriverVulkan::command_queue_execute_and_present (C:\Personal\Godot\godot_src\drivers\vulkan\rendering_device_driver_vulkan.cpp:2457)
[1] RenderingDevice::_submit_transfer_worker (C:\Personal\Godot\godot_src\servers\rendering\rendering_device.cpp:5133)
[2] RenderingDevice::_submit_transfer_workers (C:\Personal\Godot\godot_src\servers\rendering\rendering_device.cpp:5231)
[3] RenderingDevice::_end_frame (C:\Personal\Godot\godot_src\servers\rendering\rendering_device.cpp:5838)
[4] RenderingDevice::_flush_and_stall_for_all_frames (C:\Personal\Godot\godot_src\servers\rendering\rendering_device.cpp:5925)
[5] RenderingDevice::screen_prepare_for_drawing (C:\Personal\Godot\godot_src\servers\rendering\rendering_device.cpp:3722)
[6] RendererCompositorRD::blit_render_targets_to_screen (C:\Personal\Godot\godot_src\servers\rendering\renderer_rd\renderer_compositor_rd.cpp:37)
[7] RendererViewport::draw_viewports (C:\Personal\Godot\godot_src\servers\rendering\renderer_viewport.cpp:880)
[8] RenderingServerDefault::_draw (C:\Personal\Godot\godot_src\servers\rendering\rendering_server_default.cpp:88)
[9] RenderingServerDefault::draw (C:\Personal\Godot\godot_src\servers\rendering\rendering_server_default.cpp:417)
[10] Main::iteration (C:\Personal\Godot\godot_src\main\main.cpp:4410)
[11] OS_Windows::run (C:\Personal\Godot\godot_src\platform\windows\os_windows.cpp:1772)
[12] widechar_main (C:\Personal\Godot\godot_src\platform\windows\godot_windows.cpp:181)
[13] _main (C:\Personal\Godot\godot_src\platform\windows\godot_windows.cpp:206)
[14] main (C:\Personal\Godot\godot_src\platform\windows\godot_windows.cpp:220)
[15] WinMain (C:\Personal\Godot\godot_src\platform\windows\godot_windows.cpp:234)
[16] __scrt_common_main_seh (D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
[17] <couldn't map PC to fn name>
-- END OF BACKTRACE --
================================================================

Only the very first time that I had this crash, I also got this message after the backtrace log (full log):

[12992:1021/091820.075:ERROR:gpu_process_host.cc(1001)] GPU process exited unexpectedly: exit_code=34

Looking at my Event Viewer, I see two of these messages:

NVIDIA OpenGL Driver:
The NVIDIA OpenGL driver has encountered
an out of memory error. This application might
behave inconsistently and fail.
(pid=11312 godot.windows.editor.dev.x86_64.exe 64bit)

...followed by this:

Application Error:
Faulting application name: godot.windows.editor.dev.x86_64.exe, version: 4.4.0.0, time stamp: 0x67125cbd
Faulting module name: godot.windows.editor.dev.x86_64.exe, version: 4.4.0.0, time stamp: 0x67125cbd
Exception code: 0x80000003
Fault offset: 0x000000000182e1e5
Faulting process id: 0x0x2C30
Faulting application start time: 0x0x1DB23BA05E6F76F
Faulting application path: C:\Personal\Godot\godot_src\bin\godot.windows.editor.dev.x86_64.exe
Faulting module path: C:\Personal\Godot\godot_src\bin\godot.windows.editor.dev.x86_64.exe
Report Id: 863ff4d3-d526-40b6-a3a5-8eb18f3d1ed1
Faulting package full name: 
Faulting package-relative application ID: 

...and this:

OVRServiceLauncher:
The operation completed successfully. 

Before and after these events there are also a number of these messages that are likely(?) unrelated:

DeviceSetupManager:
Metadata staging failed, result=0x80070490 for container '{0E519C86-B75E-5F27-940D-77196A1CDB86}'

The OVRServiceLauncher is my Oculus Rift/Meta Quest related app that I have not used in many months, so I expect this is unrelated, but it shows that other services were affected by this NVIDIA driver crash.

I see these notices in the Event Viewer for Device Manager - NIVIDA GeForce GTX 980 Ti, likely related to the auto-update:

Information:
Driver Management has concluded the process to add Service nvlddmkm for Device Instance ID PCI\VEN_10DE&DEV_17C8&SUBSYS_32331462&REV_A1\4&256A0AA8&0&0008 with the following status: 0.
Information:
Driver Management has concluded the process to add Service NVDisplay.ContainerLocalSystem for Device Instance ID PCI\VEN_10DE&DEV_17C8&SUBSYS_32331462&REV_A1\4&256A0AA8&0&0008 with the following status: 0.

When using older versions of the editor, it doesn't crash, but instead hangs while spamming these messages:

ERROR: Condition "!dl" is true.
   at: draw_list_bind_render_pipeline (drivers/vulkan/rendering_device_vulkan.cpp:7118)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_uniform_set (drivers/vulkan/rendering_device_vulkan.cpp:7192)
ERROR: Condition "!dl" is true.
   at: draw_list_set_push_constant (drivers/vulkan/rendering_device_vulkan.cpp:7299)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_index_array (drivers/vulkan/rendering_device_vulkan.cpp:7265)
ERROR: Condition "!dl" is true.
   at: draw_list_draw (drivers/vulkan/rendering_device_vulkan.cpp:7317)
ERROR: Condition "!E" is true. Returning: TEXTURE_SAMPLES_1
   at: framebuffer_format_get_texture_samples (drivers/vulkan/rendering_device_vulkan.cpp:4116)
ERROR: Mismatch fragment shader output mask (1) and framebuffer color output mask (0) when binding both in render pipeline.
   at: (drivers/vulkan/rendering_device_vulkan.cpp:6014)
ERROR: Condition "pipeline.is_null()" is true. Returning: RID()
   at: _generate_version (servers/rendering/renderer_rd/pipeline_cache_rd.cpp:61)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_render_pipeline (drivers/vulkan/rendering_device_vulkan.cpp:7118)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_uniform_set (drivers/vulkan/rendering_device_vulkan.cpp:7192)
ERROR: Condition "!dl" is true.
   at: draw_list_set_push_constant (drivers/vulkan/rendering_device_vulkan.cpp:7299)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_index_array (drivers/vulkan/rendering_device_vulkan.cpp:7265)
ERROR: Condition "!dl" is true.
   at: draw_list_draw (drivers/vulkan/rendering_device_vulkan.cpp:7317)
ERROR: Condition "!E" is true. Returning: TEXTURE_SAMPLES_1
   at: framebuffer_format_get_texture_samples (drivers/vulkan/rendering_device_vulkan.cpp:4116)
ERROR: Mismatch fragment shader output mask (1) and framebuffer color output mask (0) when binding both in render pipeline.
   at: (drivers/vulkan/rendering_device_vulkan.cpp:6014)
ERROR: Condition "pipeline.is_null()" is true. Returning: RID()
   at: _generate_version (servers/rendering/renderer_rd/pipeline_cache_rd.cpp:61)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_render_pipeline (drivers/vulkan/rendering_device_vulkan.cpp:7118)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_uniform_set (drivers/vulkan/rendering_device_vulkan.cpp:7192)
ERROR: Condition "!dl" is true.
   at: draw_list_set_push_constant (drivers/vulkan/rendering_device_vulkan.cpp:7299)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_index_array (drivers/vulkan/rendering_device_vulkan.cpp:7265)
ERROR: Condition "!dl" is true.
   at: draw_list_draw (drivers/vulkan/rendering_device_vulkan.cpp:7317)
ERROR: Condition "!E" is true. Returning: TEXTURE_SAMPLES_1
   at: framebuffer_format_get_texture_samples (drivers/vulkan/rendering_device_vulkan.cpp:4116)
ERROR: Mismatch fragment shader output mask (1) and framebuffer color output mask (0) when binding both in render pipeline.
   at: (drivers/vulkan/rendering_device_vulkan.cpp:6014)
ERROR: Condition "pipeline.is_null()" is true. Returning: RID()
   at: _generate_version (servers/rendering/renderer_rd/pipeline_cache_rd.cpp:61)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_render_pipeline (drivers/vulkan/rendering_device_vulkan.cpp:7118)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_uniform_set (drivers/vulkan/rendering_device_vulkan.cpp:7192)
ERROR: Condition "!dl" is true.
   at: draw_list_set_push_constant (drivers/vulkan/rendering_device_vulkan.cpp:7299)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_index_array (drivers/vulkan/rendering_device_vulkan.cpp:7265)
ERROR: Condition "!dl" is true.
   at: draw_list_draw (drivers/vulkan/rendering_device_vulkan.cpp:7317)
ERROR: Condition "!E" is true. Returning: TEXTURE_SAMPLES_1
   at: framebuffer_format_get_texture_samples (drivers/vulkan/rendering_device_vulkan.cpp:4116)
ERROR: Mismatch fragment shader output mask (1) and framebuffer color output mask (0) when binding both in render pipeline.
   at: (drivers/vulkan/rendering_device_vulkan.cpp:6014)
ERROR: Condition "pipeline.is_null()" is true. Returning: RID()
   at: _generate_version (servers/rendering/renderer_rd/pipeline_cache_rd.cpp:61)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_render_pipeline (drivers/vulkan/rendering_device_vulkan.cpp:7118)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_uniform_set (drivers/vulkan/rendering_device_vulkan.cpp:7192)
ERROR: Condition "!dl" is true.
   at: draw_list_set_push_constant (drivers/vulkan/rendering_device_vulkan.cpp:7299)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_index_array (drivers/vulkan/rendering_device_vulkan.cpp:7265)
ERROR: Condition "!dl" is true.
   at: draw_list_draw (drivers/vulkan/rendering_device_vulkan.cpp:7317)
ERROR: Condition "!E" is true. Returning: TEXTURE_SAMPLES_1
   at: framebuffer_format_get_texture_samples (drivers/vulkan/rendering_device_vulkan.cpp:4116)
ERROR: Mismatch fragment shader output mask (1) and framebuffer color output mask (0) when binding both in render pipeline.
   at: (drivers/vulkan/rendering_device_vulkan.cpp:6014)
ERROR: Condition "pipeline.is_null()" is true. Returning: RID()
   at: _generate_version (servers/rendering/renderer_rd/pipeline_cache_rd.cpp:61)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_render_pipeline (drivers/vulkan/rendering_device_vulkan.cpp:7118)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_uniform_set (drivers/vulkan/rendering_device_vulkan.cpp:7192)
ERROR: Condition "!dl" is true.
   at: draw_list_set_push_constant (drivers/vulkan/rendering_device_vulkan.cpp:7299)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_index_array (drivers/vulkan/rendering_device_vulkan.cpp:7265)
ERROR: Condition "!dl" is true.
   at: draw_list_draw (drivers/vulkan/rendering_device_vulkan.cpp:7317)
ERROR: Condition "!E" is true. Returning: TEXTURE_SAMPLES_1
   at: framebuffer_format_get_texture_samples (drivers/vulkan/rendering_device_vulkan.cpp:4116)
ERROR: Mismatch fragment shader output mask (1) and framebuffer color output mask (0) when binding both in render pipeline.
   at: (drivers/vulkan/rendering_device_vulkan.cpp:6014)
ERROR: Condition "pipeline.is_null()" is true. Returning: RID()
   at: _generate_version (servers/rendering/renderer_rd/pipeline_cache_rd.cpp:61)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_render_pipeline (drivers/vulkan/rendering_device_vulkan.cpp:7118)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_uniform_set (drivers/vulkan/rendering_device_vulkan.cpp:7192)
ERROR: Condition "!dl" is true.
   at: draw_list_set_push_constant (drivers/vulkan/rendering_device_vulkan.cpp:7299)
ERROR: Condition "!dl" is true.
   at: draw_list_bind_index_array (drivers/vulkan/rendering_device_vulkan.cpp:7265)
ERROR: Condition "!dl" is true.
   at: draw_list_draw (drivers/vulkan/rendering_device_vulkan.cpp:7317)
ERROR: Condition "!E" is true. Returning: TEXTURE_SAMPLES_1
   at: framebuffer_format_get_texture_samples (drivers/vulkan/rendering_device_vulkan.cpp:4116)

Steps to reproduce

Minimal reproduction project (MRP)

Calinou commented 1 day ago

Try using a more recent commit which integrates https://github.com/godotengine/godot/pull/98086. This should fix many Vulkan synchronization issues that were causing crashes.

allenwp commented 1 day ago

Try using a more recent commit which integrates #98086. This should fix many Vulkan synchronization issues that were causing crashes.

Thanks, I'll update my branch and post back here if I experience it again after that change.

allenwp commented 1 day ago

Looking into this more, I think this crash might have been caused by Windows Update auto-updating my graphics driver in the background. I can see in the Settings app that my NVIDIA graphics driver was updated today, even though I did not do this manually: image

allenwp commented 1 day ago

Ah, I can reproduce the crash by updating my drivers:

ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_index_array (servers\rendering\rendering_device.cpp:4247)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_draw (servers\rendering\rendering_device.cpp:4313)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_uniform_set (servers\rendering\rendering_device.cpp:4176)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_render_pipeline (servers\rendering\rendering_device.cpp:4081)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_set_push_constant (servers\rendering\rendering_device.cpp:4291)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_vertex_array (servers\rendering\rendering_device.cpp:4214)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_index_array (servers\rendering\rendering_device.cpp:4247)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_draw (servers\rendering\rendering_device.cpp:4313)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_uniform_set (servers\rendering\rendering_device.cpp:4176)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_render_pipeline (servers\rendering\rendering_device.cpp:4081)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_set_push_constant (servers\rendering\rendering_device.cpp:4291)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_bind_index_array (servers\rendering\rendering_device.cpp:4247)
ERROR: Parameter "dl" is null.
   at: RenderingDevice::draw_list_draw (servers\rendering\rendering_device.cpp:4313)
ERROR: Immediate draw list is already inactive.
   at: (servers\rendering\rendering_device.cpp:4526)
ERROR: Last known breadcrumb: BLIT_PASS
   at: RenderingDeviceDriverVulkan::print_lost_device_info (drivers\vulkan\rendering_device_driver_vulkan.cpp:5120)
ERROR: VK_EXT_device_fault not available.
   at: RenderingDeviceDriverVulkan::on_device_lost (drivers\vulkan\rendering_device_driver_vulkan.cpp:4986)
ERROR: Vulkan device was lost.
   at: (drivers\vulkan\rendering_device_driver_vulkan.cpp:2456)

================================================================
CrashHandlerException: Program crashed
Engine version: Godot Engine v4.4.dev.custom_build (e501c128e3cdb0393168b3606a6f03f95bf13389)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[0] RenderingDeviceDriverVulkan::command_queue_execute_and_present (C:\Personal\Godot\godot_src\drivers\vulkan\rendering_device_driver_vulkan.cpp:2456)
[1] RenderingDevice::_submit_transfer_worker (C:\Personal\Godot\godot_src\servers\rendering\rendering_device.cpp:5134)
[2] RenderingDevice::_submit_transfer_workers (C:\Personal\Godot\godot_src\servers\rendering\rendering_device.cpp:5214)
[3] RenderingDevice::_end_frame (C:\Personal\Godot\godot_src\servers\rendering\rendering_device.cpp:5813)
[4] RenderingDevice::_flush_and_stall_for_all_frames (C:\Personal\Godot\godot_src\servers\rendering\rendering_device.cpp:5901)
[5] RenderingDevice::screen_prepare_for_drawing (C:\Personal\Godot\godot_src\servers\rendering\rendering_device.cpp:3722)
[6] RendererCompositorRD::blit_render_targets_to_screen (C:\Personal\Godot\godot_src\servers\rendering\renderer_rd\renderer_compositor_rd.cpp:37)
[7] RendererViewport::draw_viewports (C:\Personal\Godot\godot_src\servers\rendering\renderer_viewport.cpp:880)
[8] RenderingServerDefault::_draw (C:\Personal\Godot\godot_src\servers\rendering\rendering_server_default.cpp:88)
[9] RenderingServerDefault::draw (C:\Personal\Godot\godot_src\servers\rendering\rendering_server_default.cpp:417)
[10] Main::iteration (C:\Personal\Godot\godot_src\main\main.cpp:4410)
[11] OS_Windows::run (C:\Personal\Godot\godot_src\platform\windows\os_windows.cpp:1772)
[12] widechar_main (C:\Personal\Godot\godot_src\platform\windows\godot_windows.cpp:181)
[13] _main (C:\Personal\Godot\godot_src\platform\windows\godot_windows.cpp:206)
[14] main (C:\Personal\Godot\godot_src\platform\windows\godot_windows.cpp:220)
[15] WinMain (C:\Personal\Godot\godot_src\platform\windows\godot_windows.cpp:234)
[16] __scrt_common_main_seh (D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
[17] <couldn't map PC to fn name>
-- END OF BACKTRACE --
================================================================

I will update this bug report and see if I can reproduce it with the head of master.