Editor crashes on project load: "pure virtual method called"

[unfa]

Godot version

4.0 dev d11117d9d8ae32d6acc025a7495d4ae58b9c37b4

System information

Arch Linux, KDE Plasma, Radeon RX6800XT, amdgpu driver, Vulkan

Issue description

For more than a week now I have been unable to open a project I'm working on.

Godot editor crashes while loading the project. I have bisected by project commits to maybe see what in my project has triggered this engine bug, but what I found doesn't seem to make any sense, and when I tried it again to checkout the last known good commit - it crashes as well, so I don't think this will be helpful unfortunately.

What's interesting is if the assets are already imported, the game will run but not load in the editor. If the assets are not imported, the editor crashes before any of them are - the .godot/imported/ folder stays empty.

I've been testing Calinou's nightly builds every couple of days, hoping this will go away. The crashes didn't stop, but the crash messages are changing here and there.

I've been asking for help with this issue on Godot Contributors Chat. I have tried clearing all possible cache - it's one of the first things I try in such cases. I have tried removing recently created shader assets from my project since shader compilation errors were present as well. I have tried switching to single-threaded rendering, following a suggestion from @RandomShaper . I compiled Godot myself with a a patch from @bruvzg and I thought it solved the problem, but it was a just fluke.

Before finally posting here I have just compiled Godot 4.0-dev from source and tested to confirm this has not been fixed. Here's the current backtrace:

[Thread 0x7fffa8ffd6c0 (LWP 1677949) exited]

Thread 1 "godot.linuxbsd." received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f21d0) at ./core/templates/command_queue_mt.h:373
#2  0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f21d0) at ./core/templates/command_queue_mt.h:404
#3  0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f1f90, p1=...) at servers/rendering/rendering_server_default.h:916
#4  0x000055555b486010 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#5  0x000055555b4b85ca in ShaderLanguage::_parse_shader (this=0x5555619f0288, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8292
#6  0x000055555b4c54b9 in ShaderLanguage::compile (this=0x5555619f0288, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9677
#7  0x000055555b486125 in ShaderCompiler::compile (this=0x5555619f0288, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffffffc7a0, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#8  0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557bb35710, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#9  0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x5555613b9780, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#10 0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb81d1b20) at ./core/templates/command_queue_mt.h:322
#11 0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f21d0) at ./core/templates/command_queue_mt.h:373
#12 0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f21d0) at ./core/templates/command_queue_mt.h:404
#13 0x000055555b44103d in RenderingServerDefault::free (this=0x5555612f1f90, p_rid=...) at servers/rendering/rendering_server_default.h:931
#14 0x000055555a0127db in CanvasItem::~CanvasItem (this=0x5555755bb370, __in_chrg=<optimized out>) at scene/main/canvas_item.cpp:1183
#15 0x00005555582dd348 in Control::~Control (this=0x5555755bb370, __in_chrg=<optimized out>) at ./scene/gui/control.h:45
#16 0x000055555a337278 in Range::~Range (this=0x5555755bb370, __in_chrg=<optimized out>) at scene/gui/range.cpp:335
#17 0x000055555a332de4 in ProgressBar::~ProgressBar (this=0x5555755bb370, __in_chrg=<optimized out>) at scene/gui/progress_bar.h:36
#18 0x0000555557f2033f in memdelete<Node> (p_class=0x5555755bb370) at ./core/os/memory.h:109
--Type <RET> for more, q to quit, c to continue without paging--
#19 0x000055555a06ace7 in Node::_notification (this=0x5555755bbbb0, p_notification=1) at scene/main/node.cpp:169
#20 0x0000555557c1345c in Node::_notificationv (this=0x5555755bbbb0, p_notification=1, p_reversed=true) at scene/main/node.h:45
#21 0x00005555582db2ad in CanvasItem::_notificationv (this=0x5555755bbbb0, p_notification=1, p_reversed=true) at ./scene/main/canvas_item.h:45
#22 0x00005555582db9e9 in Control::_notificationv (this=0x5555755bbbb0, p_notification=1, p_reversed=true) at scene/gui/control.h:46
#23 0x000055555c0b7bbf in Object::notification (this=0x5555755bbbb0, p_notification=1, p_reversed=true) at core/object/object.cpp:735
#24 0x000055555c0b56bd in Object::_predelete (this=0x5555755bbbb0) at core/object/object.cpp:195
#25 0x000055555c0c1df3 in predelete_handler (p_object=0x5555755bbbb0) at core/object/object.cpp:1777
#26 0x0000555557f2031f in memdelete<Node> (p_class=0x5555755bbbb0) at ./core/os/memory.h:105
#27 0x000055555a06ace7 in Node::_notification (this=0x555575718480, p_notification=1) at scene/main/node.cpp:169
#28 0x0000555557c1345c in Node::_notificationv (this=0x555575718480, p_notification=1, p_reversed=true) at scene/main/node.h:45
#29 0x00005555582db2ad in CanvasItem::_notificationv (this=0x555575718480, p_notification=1, p_reversed=true) at ./scene/main/canvas_item.h:45
#30 0x00005555582db9e9 in Control::_notificationv (this=0x555575718480, p_notification=1, p_reversed=true) at scene/gui/control.h:46
#31 0x00005555582dc239 in Container::_notificationv (this=0x555575718480, p_notification=1, p_reversed=true) at ./scene/gui/container.h:37
#32 0x00005555582dc975 in BoxContainer::_notificationv (this=0x555575718480, p_notification=1, p_reversed=true) at ./scene/gui/box_container.h:37
#33 0x00005555582dd263 in HBoxContainer::_notificationv (this=0x555575718480, p_notification=1, p_reversed=true) at ./scene/gui/box_container.h:72
#34 0x000055555c0b7bbf in Object::notification (this=0x555575718480, p_notification=1, p_reversed=true) at core/object/object.cpp:735
#35 0x000055555c0b56bd in Object::_predelete (this=0x555575718480) at core/object/object.cpp:195
#36 0x000055555c0c1df3 in predelete_handler (p_object=0x555575718480) at core/object/object.cpp:1777
#37 0x000055555922dbe3 in memdelete<HBoxContainer> (p_class=0x555575718480) at ./core/os/memory.h:105
--Type <RET> for more, q to quit, c to continue without paging--c
#38 0x000055555922a9b0 in BackgroundProgress::_end_task (this=0x555569a357d0, p_task=...) at editor/progress_dialog.cpp:94
#39 0x0000555557ad1d21 in call_with_variant_args_helper<__UnexistingClass, String const&, 0ul> (p_instance=0x555569a357d0, p_method=(void (__UnexistingClass::*)(__UnexistingClass * const, const String &)) 0x55555922a8d2 <BackgroundProgress::_end_task(String const&)>, p_args=0x7fffffffd2a0, r_error=...) at ./core/variant/binder_common.h:262
#40 0x0000555557ad190e in call_with_variant_args_dv<__UnexistingClass, String const&> (p_instance=0x555569a357d0, p_method=(void (__UnexistingClass::*)(__UnexistingClass * const, const String &)) 0x55555922a8d2 <BackgroundProgress::_end_task(String const&)>, p_args=0x7fffffffd4b0, p_argcount=1, r_error=..., default_values=...) at ./core/variant/binder_common.h:409
#41 0x0000555557ad14c2 in MethodBindT<String const&>::call (this=0x555569a36410, p_object=0x555569a357d0, p_args=0x7fffffffd4b0, p_arg_count=1, r_error=...) at ./core/object/method_bind.h:320
#42 0x000055555c0b77fb in Object::callp (this=0x555569a357d0, p_method=..., p_args=0x7fffffffd4b0, p_argcount=1, r_error=...) at core/object/object.cpp:678
#43 0x000055555bd77c9c in Callable::callp (this=0x7fffe62b8038, p_arguments=0x7fffffffd4b0, p_argcount=1, r_return_value=..., r_call_error=...) at core/variant/callable.cpp:62
#44 0x000055555c0b19a0 in MessageQueue::_call_function (this=0x5555602b2880, p_callable=..., p_args=0x7fffe62b8050, p_argcount=1, p_show_error=false) at core/object/message_queue.cpp:229
#45 0x000055555c0b1cef in MessageQueue::flush (this=0x5555602b2880) at core/object/message_queue.cpp:275
#46 0x000055555a0a53aa in SceneTree::physics_process (this=0x555563bda020, p_time=0.016666666666666666) at scene/main/scene_tree.cpp:420
#47 0x0000555557ab1dea in Main::iteration () at main/main.cpp:2855
#48 0x0000555557a5a87b in OS_LinuxBSD::run (this=0x7fffffffd840) at platform/linuxbsd/os_linuxbsd.cpp:537
#49 0x0000555557a55250 in main (argc=4, argv=0x7fffffffdd68) at platform/linuxbsd/godot_linuxbsd.cpp:72

I've noticed there's a mention of instance shader uniforms, so I commented out a line in the codebase that uses that.

Now the crash occurs exactly the same way, but the output is a bit different:

Thread 1 "godot.linuxbsd." received signal SIGSEGV, Segmentation fault.
0x000055555c10ddaa in StringName::StringName (this=<error reading variable: Cannot access memory at address 0x7fffff7feff8>, p_name=<error reading variable: Cannot access memory at address 0x7fffff7feff0>, p_static=<error reading variable: Cannot access memory at address 0x7fffff7fefec>) at core/string/string_name.cpp:200
200     StringName::StringName(const char *p_name, bool p_static) {
(gdb) bt
#0  0x000055555c10ddaa in StringName::StringName (this=<error reading variable: Cannot access memory at address 0x7fffff7feff8>, p_name=<error reading variable: Cannot access memory at address 0x7fffff7feff0>, p_static=<error reading variable: Cannot access memory at address 0x7fffff7fefec>)
    at core/string/string_name.cpp:200
#1  0x000055555983dd4d in ShaderLanguage::VaryingFunctionNames::VaryingFunctionNames (this=0x7fffff7ff348) at ./servers/rendering/shader_language.h:346
#2  0x000055555983dedb in ShaderLanguage::ShaderCompileInfo::ShaderCompileInfo (this=0x7fffff7ff300) at ./servers/rendering/shader_language.h:1113
#3  0x000055555b486073 in ShaderCompiler::compile (this=0x5555619a1418, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff7ff640, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1317
#4  0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x5555797c4090, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#5  0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x5555613bd0a0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#6  0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb8336d48) at ./core/templates/command_queue_mt.h:322
#7  0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:373
#8  0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:404
#9  0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f5f10, p1=...) at servers/rendering/rendering_server_default.h:916
#10 0x000055555b486010 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#11 0x000055555b4b85ca in ShaderLanguage::_parse_shader (this=0x5555619a1418, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8292
#12 0x000055555b4c54b9 in ShaderLanguage::compile (this=0x5555619a1418, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9677
#13 0x000055555b486125 in ShaderCompiler::compile (this=0x5555619a1418, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff800650, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#14 0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x5555797c4090, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#15 0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x5555613bd0a0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#16 0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb8336d48) at ./core/templates/command_queue_mt.h:322
#17 0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:373
#18 0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:404
#19 0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f5f10, p1=...) at servers/rendering/rendering_server_default.h:916
#20 0x000055555b486010 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#21 0x000055555b4b85ca in ShaderLanguage::_parse_shader (this=0x5555619a1418, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8292
#22 0x000055555b4c54b9 in ShaderLanguage::compile (this=0x5555619a1418, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9677
#23 0x000055555b486125 in ShaderCompiler::compile (this=0x5555619a1418, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff801660, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#24 0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x5555797c4090, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#25 0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x5555613bd0a0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#26 0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb8336d48) at ./core/templates/command_queue_mt.h:322
#27 0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:373
#28 0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:404
#29 0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f5f10, p1=...) at servers/rendering/rendering_server_default.h:916
#30 0x000055555b486010 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#31 0x000055555b4b85ca in ShaderLanguage::_parse_shader (this=0x5555619a1418, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8292
#32 0x000055555b4c54b9 in ShaderLanguage::compile (this=0x5555619a1418, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9677
#33 0x000055555b486125 in ShaderCompiler::compile (this=0x5555619a1418, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff802670, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#34 0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x5555797c4090, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#35 0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x5555613bd0a0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#36 0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb8336d48) at ./core/templates/command_queue_mt.h:322
#37 0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:373
#38 0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:404
#39 0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f5f10, p1=...) at servers/rendering/rendering_server_default.h:916
#40 0x000055555b486010 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#41 0x000055555b4b85ca in ShaderLanguage::_parse_shader (this=0x5555619a1418, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8292
#42 0x000055555b4c54b9 in ShaderLanguage::compile (this=0x5555619a1418, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9677
#43 0x000055555b486125 in ShaderCompiler::compile (this=0x5555619a1418, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff803680, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#44 0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x5555797c4090, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#45 0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x5555613bd0a0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#46 0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb8336d48) at ./core/templates/command_queue_mt.h:322
#47 0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:373
#48 0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:404
#49 0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f5f10, p1=...) at servers/rendering/rendering_server_default.h:916
(...)

This goes on for 20k lines now. Here's the full thing: full backtrace.zip

I've also tried doing an completely clean git repo clone, as in the past that has proven to "solve" some issues that nothing else helped with. How does that work - I have absolutely no idea, but it does. I guess sometimes a repo clone on my disk just gets cursed and I need to get a clean one, checkout where I was and keep going :stuck_out_tongue_closed_eyes:

So this is where I am at the moment.

Steps to reproduce

If you'd like to test my project on your own (unfortunately it's definitely not a minimal one - clean repo clone is 2.2 GB right now), here's how:

1. Clone my project repo: https://codeberg.org/unfa/Liblast
2. Initialize LFS: git lfs install (it seems there is no need to do git lfs pull but it won't hurt).
3. Check out server-refactor branch
4. Try to open the project located in Liblast/Game in the Godot editor.

[jordo]

@unfa The second crashdump is a stack memory error. Considering your stack in the zip is 20392 frames long... it's likely your program runs out of stack memory and segfaults. The first crashdump would be well served to provide a crashdump of an asan build. I think @Calinou provides asan editor builds. That segfault could be anything, but with asan at least we can see what type of memory error it is as asan will quarantine memory regions and instrument every load and store. It can then report the type of memory error, place of allocation, free, etc. A bunch of useful things.

[jordo]

#1  0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f21d0) at ./core/templates/command_queue_mt.h:373
#2  0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f21d0) at ./core/templates/command_queue_mt.h:404
#3  0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f1f90, p1=...) at servers/rendering/rendering_server_default.h:916
#4  0x000055555b486010 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#5  0x000055555b4b85ca in ShaderLanguage::_parse_shader (this=0x5555619f0288, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8292
#6  0x000055555b4c54b9 in ShaderLanguage::compile (this=0x5555619f0288, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9677
#7  0x000055555b486125 in ShaderCompiler::compile (this=0x5555619f0288, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffffffc7a0, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#8  0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557bb35710, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#9  0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x5555613b9780, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#10 0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb81d1b20) at ./core/templates/command_queue_mt.h:322
#11 0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f21d0) at ./core/templates/command_queue_mt.h:373
#12 0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f21d0) at ./core/templates/command_queue_mt.h:404
#13 0x000055555b44103d in RenderingServerDefault::free (this=0x5555612f1f90, p_rid=...) at servers/rendering/rendering_server_default.h:931
#14 0x000055555a0127db in CanvasItem::~CanvasItem (this=0x5555755bb370, __in_chrg=<optimized out>) at scene/main/canvas_item.cpp:1183

I'm not up to date with 4.X, but it looks odd that canvas item destructor CanvasItem::~CanvasItem eventually calls through to RendererRD::MaterialStorage::shader_set_code that really seems not correct. I guess it's causing CommandQueue to flush? Seems odd.

At any rate, the first crashdump, has BackgroundProgress::_end_task, so could be some memory issue caused by thread issue. Asan build will report helpful information when memory access violation happens.

[Rubonnek]

I was able to reproduce the issue. Here's the ASAN report: asan_report.txt

The reports shows an use-after-free, but in particular what I'm seeing with gdb is that there's a CommandBase casted object that seems to get overwritten at some point. It's function pointer at cmd->call gets zeroed out and that's what's causing a segfault:

Thread 1 "godot-master" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) up
#1  0x0000555555f4f9ac in CommandQueueMT::_flush (this=0x55556287fac0) at ./core/templates/command_queue_mt.h:373
373                             cmd->call(); //execute the function

[jordo]

Your asan report looks like a standard use after free, combined with multithreading issue and focus there... Use after free can continue on in certain circumstances and continue to cause more memory corruption which can lead to weird things happening, so start with fixing the use after free memory error reported by asan before debugging further with gdb in non-asan builds.

From your asan attached report: 1): The engine is allocing a GradientTexture1D ON THREAD T50 here: (This thread looks like it is a EditorFileSystem::scan thread).

#1 0x561ce6be6538 in Memory::alloc_static(unsigned long, bool) core/os/memory.cpp:75
    #2 0x561ce6be6456 in operator new(unsigned long, char const*) core/os/memory.cpp:40
    #3 0x561cde94889a in Object* ClassDB::creator<GradientTexture1D>() (/opt/godot-master/bin/godot.linuxbsd.opt.tools.64.san+0xa44389a)
    #4 0x561ce825cbbb in ClassDB::instantiate(StringName const&) core/object/class_db.cpp:338
    #5 0x561ce1af3537 in ResourceLoaderText::load() scene/resources/resource_format_text.cpp:565

2): It is being free on the same editor scan thread (T50) here:

    #1 0x561ce6be710e in Memory::free_static(void*, bool) core/os/memory.cpp:168
    #2 0x561cd5d4dfc2 in void memdelete<Texture2D>(Texture2D*) core/os/memory.h:112
    #3 0x561cd5d44347 in Ref<Texture2D>::unref() core/object/ref_counted.h:221
    #4 0x561cd5d38f79 in Ref<Texture2D>::~Ref() core/object/ref_counted.h:233
    #5 0x561ce19a53a3 in ParticlesMaterial::~ParticlesMaterial() scene/resources/particles_material.cpp:1867

3): And it is being used again on main thread (T0) here:

    #0 0x561ce203577f in GradientTexture1D::_update() scene/resources/texture.cpp:2240
    #1 0x561cd6213be7 in void call_with_variant_args_helper<__UnexistingClass>(__UnexistingClass*, void (__UnexistingClass::*)(), Variant const**, Callable::CallError&, IndexSequence<>) core/variant/binder_common.h:262
    #2 0x561cd6204492 in void call_with_variant_args_dv<__UnexistingClass>(__UnexistingClass*, void (__UnexistingClass::*)(), Variant const**, int, Callable::CallError&, Vector<Variant> const&) core/variant/binder_common.h:409
    #3 0x561cd61f0d7e in MethodBindT<>::call(Object*, Variant const**, int, Callable::CallError&) core/object/method_bind.h:320
    #4 0x561ce82cb5cd in Object::callp(StringName const&, Variant const**, int, Callable::CallError&) core/object/object.cpp:678
    #5 0x561ce74c5457 in Callable::callp(Variant const**, int, Variant&, Callable::CallError&) const core/variant/callable.cpp:62
    #6 0x561ce82ae53c in MessageQueue::_call_function(Callable const&, Variant const*, int, bool) core/object/message_queue.cpp:229
    #7 0x561ce82aef6b in MessageQueue::flush() core/object/message_queue.cpp:275

So some callable, to GradientTexture1D::_update( is queued in the MessageQueue for processing on the main thread, which is executing on the GradientTexture1D (which was already freed from the originating T50 thread).

In your particular case, it looks to be caused by something in a particles material which can at least start to point you in some direction:

#5 0x561ce19a53a3 in ParticlesMaterial::~ParticlesMaterial() scene/resources/particles_material.cpp:1867

[jordo]

So.... I'm almost willing to bet, this is the cause (in texture.cpp):

If this queue_update API is called on an object from ANY non-main thread, and the object is destructed before the MessageQueue on the main thread is flushed (call_deferred pushes that onto this queue for processing), guess what? Kablam...

I really have a problem with the call_deferred design, because it's really hacky and easily leads to bugs like this which hinders the ability to do multithreaded things in the engine safely.

I'm not sure what the specific fix is for this issue, but overall call_deferred should probably warn or do something when it's called from a non-main thread, or if it's used from a Resource or any reference counted object (Ref<>), increment the reference count (delays object destruction), and decrement after callable is flushed from the main thread's MessageQueue. It's always dangerous pushing something like an Object* across threads into an deferred async processing queue which has no pointer ownership or sharing semantics... But will be up to engine maintainers to decide how to proceed with any overall design change.

[jordo]

Ahhh, ok. So digging a bit further, here is potential root source of error (still related to call_deferred).... (This is also why thread safety is tricky to get right).

So Callable doesn't actually store (well in most cases) an Object*, but an Object ID. Which is OK as it does a lookup to retrieve the object by ID before actually jumping to the callable exec (see below screencap).

However, the lines between the object retrieval and the exec function on the object (line 52 and line 62), are not thread-safe. I.e. line 52 ObjectDB::get_instance(ObjectID(object)) can retrieve a valid Object but the Object could be freed by another thread before line 62 is executed. (We know this particular issue has object destruction on a different thread as reported by asan).

The correct solution for this is not that trivial as it has implication for how Callable's work, and for how locking in the ObjectDB is done. But, the current design is not thread-safe. To make Callable's thread-safe, they need to ensure the lifetime of the object extends past actually executing obj->callp (line 62).

[jordo]

My suggestion would be to start migrating away from using raw Object* (which will continue to have dangling pointer issues due to lack of any ownership semantics), and move into using something like a std::weak_ptr, which can be tested and converted into a strong reference before usage, ex before: (obj->callp) which will guarantee lifetime beyond the function call.

An async processed MessageQueue (godot's main thread message queue) would be a good example of where something like std::weak_ptr is a good design choice for how the queue stores the actual object pointers in it's internal callable queue. This would also make callable's faster, as there would be no lookup cost and no global locking (there is a global spinlock in ObjectDB::get_instance).

[RandomShaper]

This comes to mind: https://github.com/godotengine/godot/pull/40764.

[Calinou]

I think @Calinou provides asan editor builds.

I don't – https://github.com/Calinou/godot-debug-builds only contains standard debug symbols. Sanitizers typically require using recent compilers, which compromise support of old Linux distributions. This in turn makes those binaries difficult to distribute publicly.

Once 4.0 is out, it's probably fine to use Ubuntu 20.04 has a baseline (which should have decent sanitizers in its packaged compilers). Until then, I'd prefer sticking to Ubuntu 18.04 as a baseline.

[derammo]

If this queue_update API is called on an object from ANY non-main thread, and the object is destructed before the MessageQueue on the main thread is flushed (call_deferred pushes that onto this queue for processing), guess what? Kablam...

That's just a straight up bug. You can't synchronously destroy an object that participates in async callbacks. You have to mark the object as "out of service, don't make any more async requests" and defer the destruction itself to get it to sort after any other async operations. I think this is just another case of people using call_deferred as a cheap thread safety hack, which doesn't work unless your object has eternal (i.e. scene graph) lifetime.

[Edit: to clarify, it is totally fine to do this, but you do have to make sure your object outlives the next iteration. So, in this case if a scanner is temporarily loading a resource, that's not appropriate.]

[Another edit: Yes, I understand the soft reference (Object ID) handles this for the main thread by itself (by failing all the async calls), but it just doesn't for multi-threading, as you pointed out above.]

[derammo]

Oh and regarding the excellent std::weak_ptr recommendation, you're basically just asking for call_deferred to be moved to RefCounted. That would actually implement the current rules that the Object's lifetime has to be ensured if it is used across threads. That makes tons of sense to me, but for some design reason, call_deferred is allowed on Object. It would be helpful if an architect could explain why that is important. Somewhere there might be a zillion non counted objects that need it?

[unfa]

BTW, I've reverted a problematic commit in my project that seems to be the first bad one (after the codebase) and it didn't fix the problem, but the backtrace looks different:

[Thread 0x7fffb0ff96c0 (LWP 1708026) exited]

Thread 1 "godot.linuxbsd." received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f8a30) at ./core/templates/command_queue_mt.h:373
#2  0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f8a30) at ./core/templates/command_queue_mt.h:404
#3  0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f87f0, p1=...) at servers/rendering/rendering_server_default.h:916
#4  0x000055555b486010 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#5  0x000055555b4b85ca in ShaderLanguage::_parse_shader (this=0x5555619a63b8, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8292
#6  0x000055555b4c54b9 in ShaderLanguage::compile (this=0x5555619a63b8, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9677
#7  0x000055555b486125 in ShaderCompiler::compile (this=0x5555619a63b8, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffffffc7a0, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#8  0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557a485b00, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#9  0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x555561333580, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#10 0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb817ce90) at ./core/templates/command_queue_mt.h:322
#11 0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f8a30) at ./core/templates/command_queue_mt.h:373
#12 0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f8a30) at ./core/templates/command_queue_mt.h:404
#13 0x000055555b44103d in RenderingServerDefault::free (this=0x5555612f87f0, p_rid=...) at servers/rendering/rendering_server_default.h:931
#14 0x000055555a0127db in CanvasItem::~CanvasItem (this=0x55557577fea0, __in_chrg=<optimized out>) at scene/main/canvas_item.cpp:1183
#15 0x00005555582dd348 in Control::~Control (this=0x55557577fea0, __in_chrg=<optimized out>) at ./scene/gui/control.h:45
#16 0x000055555a337278 in Range::~Range (this=0x55557577fea0, __in_chrg=<optimized out>) at scene/gui/range.cpp:335
#17 0x000055555a332de4 in ProgressBar::~ProgressBar (this=0x55557577fea0, __in_chrg=<optimized out>) at scene/gui/progress_bar.h:36
#18 0x0000555557f2033f in memdelete<Node> (p_class=0x55557577fea0) at ./core/os/memory.h:109
--Type <RET> for more, q to quit, c to continue without paging--
#19 0x000055555a06ace7 in Node::_notification (this=0x5555757806e0, p_notification=1) at scene/main/node.cpp:169
#20 0x0000555557c1345c in Node::_notificationv (this=0x5555757806e0, p_notification=1, p_reversed=true) at scene/main/node.h:45
#21 0x00005555582db2ad in CanvasItem::_notificationv (this=0x5555757806e0, p_notification=1, p_reversed=true) at ./scene/main/canvas_item.h:45
#22 0x00005555582db9e9 in Control::_notificationv (this=0x5555757806e0, p_notification=1, p_reversed=true) at scene/gui/control.h:46
#23 0x000055555c0b7bbf in Object::notification (this=0x5555757806e0, p_notification=1, p_reversed=true) at core/object/object.cpp:735
#24 0x000055555c0b56bd in Object::_predelete (this=0x5555757806e0) at core/object/object.cpp:195
#25 0x000055555c0c1df3 in predelete_handler (p_object=0x5555757806e0) at core/object/object.cpp:1777
#26 0x0000555557f2031f in memdelete<Node> (p_class=0x5555757806e0) at ./core/os/memory.h:105
#27 0x000055555a06ace7 in Node::_notification (this=0x555574496a50, p_notification=1) at scene/main/node.cpp:169
#28 0x0000555557c1345c in Node::_notificationv (this=0x555574496a50, p_notification=1, p_reversed=true) at scene/main/node.h:45
#29 0x00005555582db2ad in CanvasItem::_notificationv (this=0x555574496a50, p_notification=1, p_reversed=true) at ./scene/main/canvas_item.h:45
#30 0x00005555582db9e9 in Control::_notificationv (this=0x555574496a50, p_notification=1, p_reversed=true) at scene/gui/control.h:46
#31 0x00005555582dc239 in Container::_notificationv (this=0x555574496a50, p_notification=1, p_reversed=true) at ./scene/gui/container.h:37
#32 0x00005555582dc975 in BoxContainer::_notificationv (this=0x555574496a50, p_notification=1, p_reversed=true) at ./scene/gui/box_container.h:37
#33 0x00005555582dd263 in HBoxContainer::_notificationv (this=0x555574496a50, p_notification=1, p_reversed=true) at ./scene/gui/box_container.h:72
#34 0x000055555c0b7bbf in Object::notification (this=0x555574496a50, p_notification=1, p_reversed=true) at core/object/object.cpp:735
#35 0x000055555c0b56bd in Object::_predelete (this=0x555574496a50) at core/object/object.cpp:195
#36 0x000055555c0c1df3 in predelete_handler (p_object=0x555574496a50) at core/object/object.cpp:1777
#37 0x000055555922dbe3 in memdelete<HBoxContainer> (p_class=0x555574496a50) at ./core/os/memory.h:105
#38 0x000055555922a9b0 in BackgroundProgress::_end_task (this=0x555569a47020, p_task=...) at editor/progress_dialog.cpp:94
#39 0x0000555557ad1d21 in call_with_variant_args_helper<__UnexistingClass, String const&, 0ul> (p_instance=0x555569a47020, 
    p_method=(void (__UnexistingClass::*)(__UnexistingClass * const, const String &)) 0x55555922a8d2 <BackgroundProgress::_end_task(String const&)>, p_args=0x7fffffffd2a0, r_error=...) at ./core/variant/binder_common.h:262
#40 0x0000555557ad190e in call_with_variant_args_dv<__UnexistingClass, String const&> (p_instance=0x555569a47020, 
    p_method=(void (__UnexistingClass::*)(__UnexistingClass * const, const String &)) 0x55555922a8d2 <BackgroundProgress::_end_task(String const&)>, p_args=0x7fffffffd4b0, p_argcount=1, r_error=..., default_values=...)
    at ./core/variant/binder_common.h:409
#41 0x0000555557ad14c2 in MethodBindT<String const&>::call (this=0x555569a47c60, p_object=0x555569a47020, p_args=0x7fffffffd4b0, p_arg_count=1, r_error=...) at ./core/object/method_bind.h:320
#42 0x000055555c0b77fb in Object::callp (this=0x555569a47020, p_method=..., p_args=0x7fffffffd4b0, p_argcount=1, r_error=...) at core/object/object.cpp:678
#43 0x000055555bd77c9c in Callable::callp (this=0x7fffe62b8038, p_arguments=0x7fffffffd4b0, p_argcount=1, r_return_value=..., r_call_error=...) at core/variant/callable.cpp:62
#44 0x000055555c0b19a0 in MessageQueue::_call_function (this=0x5555602b2880, p_callable=..., p_args=0x7fffe62b8050, p_argcount=1, p_show_error=false) at core/object/message_queue.cpp:229
#45 0x000055555c0b1cef in MessageQueue::flush (this=0x5555602b2880) at core/object/message_queue.cpp:275
#46 0x000055555a0a53aa in SceneTree::physics_process (this=0x555563bcdbb0, p_time=0.016666666666666666) at scene/main/scene_tree.cpp:420
#47 0x0000555557ab1dea in Main::iteration () at main/main.cpp:2855
#48 0x0000555557a5a87b in OS_LinuxBSD::run (this=0x7fffffffd840) at platform/linuxbsd/os_linuxbsd.cpp:537
(...)

[unfa]

I've gone back to the last known good commit in my project and branched off form there, hoping I can do some work done before this is fixed.

I've compiled Godot from 106b6805018649b13da9e9508e80611f62ed660a and carried on.

Then I have had a random crash while working on some UI elements, I am not sure if it's related, but a lot in this backtrace seems familiar:

handle_crash: Program crashed with signal 11
Engine version: Godot Engine v4.0.alpha.custom_build (106b6805018649b13da9e9508e80611f62ed660a)
Dumping the backtrace. Please include this when reporting the bug on: https://github.com/godotengine/godot/issues
[1] /usr/lib/libc.so.6(+0x38a40) [0x7ffff7ac6a40] (??:0)
[2] CompressedTexture2D::~CompressedTexture2D() (??:0)
[3] void memdelete<RefCounted>(RefCounted*) (??:0)
[4] Variant::_clear_internal() (??:0)
[5] Variant::clear() (??:0)
[6] Variant::~Variant() (??:0)
[7] CowData<Variant>::_unref(void*) (??:0)
[8] CowData<Variant>::~CowData() (??:0)
[9] Vector<Variant>::~Vector() (??:0)
[10] KeyValue<DisplayServer::CursorShape, Vector<Variant> >::~KeyValue() (??:0)
[11] HashMapElement<DisplayServer::CursorShape, Vector<Variant> >::~HashMapElement() (??:0)
[12] void memdelete<HashMapElement<DisplayServer::CursorShape, Vector<Variant> > >(HashMapElement<DisplayServer::CursorShape, Vector<Variant> >*) (??:0)
[13] DefaultTypedAllocator<HashMapElement<DisplayServer::CursorShape, Vector<Variant> > >::delete_allocation(HashMapElement<DisplayServer::CursorShape, Vector<Variant> >*) (??:0)
[14] HashMap<DisplayServer::CursorShape, Vector<Variant>, HashMapHasherDefault, HashMapComparatorDefault<DisplayServer::CursorShape>, DefaultTypedAllocator<HashMapElement<DisplayServer::CursorShape, Vector<Variant> > > >::clear() (??:0)
[15] HashMap<DisplayServer::CursorShape, Vector<Variant>, HashMapHasherDefault, HashMapComparatorDefault<DisplayServer::CursorShape>, DefaultTypedAllocator<HashMapElement<DisplayServer::CursorShape, Vector<Variant> > > >::~HashMap() (??:0)
[16] DisplayServerX11::~DisplayServerX11() (??:0)
[17] void memdelete<DisplayServer>(DisplayServer*) (??:0)
[18] finalize_display() (??:0)
[19] Main::cleanup(bool) (??:0)
[20] /data/Projects/Games/Liblast/Repo/Godot/godot-repo/bin/godot.linuxbsd.tools.64(main+0x191) [0x555557a5525a] (??:0)
[21] /usr/lib/libc.so.6(+0x232d0) [0x7ffff7ab12d0] (??:0)
[22] /usr/lib/libc.so.6(__libc_start_main+0x8a) [0x7ffff7ab138a] (??:0)
[23] /data/Projects/Games/Liblast/Repo/Godot/godot-repo/bin/godot.linuxbsd.tools.64(_start+0x25) [0x555557a54ff5] (??:0)
-- END OF BACKTRACE --
================================================================
(gdb) bt
#0  0x00007ffff7b17994 in pthread_mutex_lock () from /usr/lib/libc.so.6
#1  0x0000555557a6276a in __gthread_mutex_lock (__mutex=0x62006100720255) at /usr/include/c++/12.1.1/x86_64-pc-linux-gnu/bits/gthr-default.h:749
#2  0x0000555557a627ba in __gthread_recursive_mutex_lock (__mutex=0x62006100720255) at /usr/include/c++/12.1.1/x86_64-pc-linux-gnu/bits/gthr-default.h:811
#3  0x0000555557a627ee in std::recursive_mutex::lock (this=0x62006100720255) at /usr/include/c++/12.1.1/mutex:108
#4  0x000055555a0adb56 in MutexImpl<std::recursive_mutex>::lock (this=0x62006100720255) at ./core/os/mutex.h:47
#5  MutexLock<MutexImpl<std::recursive_mutex> >::MutexLock (p_mutex=..., this=<optimized out>) at ./core/os/mutex.h:66
#6  SceneTree::queue_delete (this=0x62006100720065, p_object=0x55557d3cd500) at scene/main/scene_tree.cpp:1075
#7  0x000055555a07f622 in Node::queue_delete (this=0x55557d3cd500) at scene/main/node.cpp:2628
#8  0x000055555a0d9344 in Viewport::_gui_cancel_tooltip (this=0x555563329a00) at scene/main/viewport.cpp:1127
#9  0x000055555a0db2b7 in Viewport::_gui_input_event (this=0x555563329a00, p_event=...) at scene/main/viewport.cpp:1552
#10 0x000055555a0e2a15 in Viewport::push_input (this=0x555563329a00, p_event=..., p_local_coords=false) at scene/main/viewport.cpp:2746
#11 0x000055555a124667 in Window::_window_input (this=0x555563329a00, p_ev=...) at scene/main/window.cpp:1045
#12 0x000055555a144023 in call_with_variant_args_helper<Window, Ref<InputEvent> const&, 0ul> (p_instance=0x555563329a00, p_method=(void (Window::*)(Window * const, const Ref<InputEvent> &)) 0x55555a1244ba <Window::_window_input(Ref<InputEvent> const&)>, p_args=0x7fffffffd248, r_error=...)
    at ./core/variant/binder_common.h:262
#13 0x000055555a1410f9 in call_with_variant_args<Window, Ref<InputEvent> const&> (p_instance=0x555563329a00, p_method=(void (Window::*)(Window * const, const Ref<InputEvent> &)) 0x55555a1244ba <Window::_window_input(Ref<InputEvent> const&)>, p_args=0x7fffffffd248, p_argcount=1, r_error=...)
    at ./core/variant/binder_common.h:376
#14 0x000055555a13d2e8 in CallableCustomMethodPointer<Window, Ref<InputEvent> const&>::call (this=0x5555705789b0, p_arguments=0x7fffffffd248, p_argcount=1, r_return_value=..., r_call_error=...) at ./core/object/callable_method_pointer.h:104
#15 0x000055555bd7d2b1 in Callable::callp (this=0x7fffffffd290, p_arguments=0x7fffffffd248, p_argcount=1, r_return_value=..., r_call_error=...) at core/variant/callable.cpp:50
#16 0x0000555557a75d36 in DisplayServerX11::_dispatch_input_event (this=0x5555606e4f00, p_event=...) at platform/linuxbsd/display_server_x11.cpp:3184
#17 0x0000555557a75a9d in DisplayServerX11::_dispatch_input_events (p_event=...) at platform/linuxbsd/display_server_x11.cpp:3155
#18 0x000055555bd1efdb in Input::_parse_input_event_impl (this=0x55556062a9d0, p_event=..., p_is_emulated=false) at core/input/input.cpp:663
#19 0x000055555bd20507 in Input::flush_buffered_events (this=0x55556062a9d0) at core/input/input.cpp:888
#20 0x0000555557a7a94c in DisplayServerX11::process_events (this=0x5555606e4f00) at platform/linuxbsd/display_server_x11.cpp:4173
#21 0x0000555557a5a887 in OS_LinuxBSD::run (this=0x7fffffffd840) at platform/linuxbsd/os_linuxbsd.cpp:536
#22 0x0000555557a55250 in main (argc=4, argv=0x7fffffffdd68) at platform/linuxbsd/godot_linuxbsd.cpp:72

[rcorre]

I've hit the exact stacktrace above (https://github.com/godotengine/godot/issues/64190#issuecomment-1211419628) several times in Godot 4. I'm not able to consistently repro. https://github.com/godotengine/godot/issues/63555 and https://github.com/godotengine/godot/issues/59798 also mention Viewport::_gui_cancel_tooltip

[unfa]

TL;DR One can open the project in Godot editor from the last good commit, then check out the bad commit, reload scenes and scripts and carry on. The crash will only happen after the project is fully unloaded and loaded again.

Wider context:

To not be completely blocked in my project≤ I've branched off from the last good commit, and carried on with some other work. I've tried merging the rest of the problematic branch back into my new one to see if this problem will persist.

It did, but I've noticed something interesting - the crash occurred only after I've manually reloaded the whole project.

When I've just finished my merge at first Godot asked to reload scenes that were changed on disk, I did that and was able to keep working on the project and run the game.

Only after I've reloaded the project the crash has happened. So there is something that's being called only on initial project load that causes the crash. This seems to me like something rather simple, as it clearly is possible to still work on the project and run it if the initial loading is skipped. It seems that I could "work this around" by always checking my known last good commit, opening the project in Godot and then checkout the branch to update the project.

[unfa]

After I rolled back my merge with git reset --soft HEAD~1 and then did git rest --hard HEAD I still have a crash on project load, but it's a bit different:

#0  0x000055555bde9b50 in StringName::StringName (this=<error reading variable: Cannot access memory at address 0x7fffff7fefe8>, p_name=<error reading variable: Cannot access memory at address 0x7fffff7fefe0>, 
    p_static=<error reading variable: Cannot access memory at address 0x7fffff7fefdc>) at core/string/string_name.cpp:200
#1  0x00005555595fddfe in ShaderLanguage::VaryingFunctionNames::VaryingFunctionNames (this=0x7fffff7ff348) at ./servers/rendering/shader_language.h:346
#2  0x00005555595fdf87 in ShaderLanguage::ShaderCompileInfo::ShaderCompileInfo (this=0x7fffff7ff300) at ./servers/rendering/shader_language.h:1113
#3  0x000055555b197ba5 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff7ff6a0, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1317
#4  0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#5  0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#6  0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#7  0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#8  0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#9  0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916
#10 0x000055555b197b42 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#11 0x000055555b1c97a4 in ShaderLanguage::_parse_shader (this=0x555561574d48, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8330
#12 0x000055555b1d6417 in ShaderLanguage::compile (this=0x555561574d48, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#13 0x000055555b197c57 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff800710, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#14 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#15 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#16 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#17 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#18 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#19 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916
#20 0x000055555b197b42 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#21 0x000055555b1c97a4 in ShaderLanguage::_parse_shader (this=0x555561574d48, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8330
#22 0x000055555b1d6417 in ShaderLanguage::compile (this=0x555561574d48, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#23 0x000055555b197c57 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff801780, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#24 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#25 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#26 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#27 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#28 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#29 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916
#30 0x000055555b197b42 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#31 0x000055555b1c97a4 in ShaderLanguage::_parse_shader (this=0x555561574d48, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8330
#32 0x000055555b1d6417 in ShaderLanguage::compile (this=0x555561574d48, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#33 0x000055555b197c57 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff8027f0, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#34 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#35 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#36 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#37 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#38 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#39 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916
#40 0x000055555b197b42 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#41 0x000055555b1c97a4 in ShaderLanguage::_parse_shader (this=0x555561574d48, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8330
#42 0x000055555b1d6417 in ShaderLanguage::compile (this=0x555561574d48, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#43 0x000055555b197c57 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff803860, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#44 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#45 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#46 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#47 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#48 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#49 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916

This goes on for 20k lines and ends with:

#19894 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#19895 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#19896 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#19897 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#19898 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#19899 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916
#19900 0x000055555b197b42 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#19901 0x000055555b1c97a4 in ShaderLanguage::_parse_shader (this=0x555561574d48, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8330
#19902 0x000055555b1d6417 in ShaderLanguage::compile (this=0x555561574d48, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#19903 0x000055555b197c57 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffffffbd40, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#19904 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#19905 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#19906 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#19907 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#19908 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#19909 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916
#19910 0x000055555b197b42 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#19911 0x000055555b1c97a4 in ShaderLanguage::_parse_shader (this=0x555561574d48, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8330
#19912 0x000055555b1d6417 in ShaderLanguage::compile (this=0x555561574d48, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#19913 0x000055555b197c57 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffffffcdb0, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#19914 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#19915 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#19916 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#19917 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#19918 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#19919 0x000055555b13d965 in RenderingServerDefault::material_set_shader (this=0x555560fca7c0, p1=..., p2=...) at servers/rendering/rendering_server_default.h:242
#19920 0x000055555a89a10f in BaseMaterial3D::_update_shader (this=0x7fffbc0aa940) at scene/resources/material.cpp:1438
#19921 0x000055555a89aab9 in BaseMaterial3D::flush_changes () at scene/resources/material.cpp:1445
#19922 0x0000555559e1bbf8 in SceneTree::_call_idle_callbacks (this=0x555563fd6330) at scene/main/scene_tree.cpp:1344
#19923 0x0000555559e13c07 in SceneTree::process (this=0x555563fd6330, p_time=0.016719999999999999) at scene/main/scene_tree.cpp:500
#19924 0x0000555557940bed in Main::iteration () at main/main.cpp:2885
#19925 0x00005555578f5dce in OS_LinuxBSD::run (this=0x7fffffffd780) at platform/linuxbsd/os_linuxbsd.cpp:540
#19926 0x00005555578f1260 in main (argc=4, argv=0x7fffffffdca8) at platform/linuxbsd/godot_linuxbsd.cpp:72

I guess we can see where this endless loop has started. Is it #19921 0x000055555a89aab9 in BaseMaterial3D::flush_changes () at scene/resources/material.cpp:1445 ? Is that a hint?

I wonder why this persists even though code on my disk is now rolled back...

[unfa]

I've moved the .godot folder hoping I could get back this branch to a working state but I still get the same crash, it just ends the loop on a different line of code:

Thread 1 "godot" received signal SIGSEGV, Segmentation fault.
0x000055555be06eac in String::operator+= (this=<error reading variable: Cannot access memory at address 0x7fffff7fefe8>, p_char=<error reading variable: Cannot access memory at address 0x7fffff7fefe4>) at core/string/ustring.cpp:583
583     core/string/ustring.cpp: No such file or directory.
(gdb) bt
#0  0x000055555be06eac in String::operator+= (this=<error reading variable: Cannot access memory at address 0x7fffff7fefe8>, p_char=<error reading variable: Cannot access memory at address 0x7fffff7fefe4>) at core/string/ustring.cpp:583
#1  0x000055555b1a03cb in ShaderLanguage::_get_token (this=0x55556148ade8) at servers/rendering/shader_language.cpp:805
#2  0x000055555b1c58c5 in ShaderLanguage::_parse_shader (this=0x55556148ade8, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:7803
#3  0x000055555b1d6417 in ShaderLanguage::compile (this=0x55556148ade8, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#4  0x000055555b197c57 in ShaderCompiler::compile (this=0x55556148ade8, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff7ffc90, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#5  0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557611bab0, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#6  0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x555561097fe0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#7  0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffac13b558) at ./core/templates/command_queue_mt.h:322
#8  0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fbc970) at ./core/templates/command_queue_mt.h:373
#9  0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fbc970) at ./core/templates/command_queue_mt.h:404
#10 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fbc730, p1=...) at servers/rendering/rendering_server_default.h:916

[Zireael07]

Is this a crash or a freeze? I've had a freeze on project open that required xkill, so likely an infinite loop somewhere in Godot's code

[bruvzg]

I guess we can see where this endless loop has started. Is it #19921 0x000055555a89aab9 in BaseMaterial3D::flush_changes () at scene/resources/material.cpp:1445 ? Is that a hint?

Not sure what's wrong with BaseMaterial3D::flush_changes, but if it's calling flush recursively, try something like:

diff --git a/core/templates/command_queue_mt.h b/core/templates/command_queue_mt.h
index 7d3e31b5bc..37cdd7d1f5 100644
--- a/core/templates/command_queue_mt.h
+++ b/core/templates/command_queue_mt.h
@@ -340,9 +340,13 @@ class CommandQueueMT {
    SyncSemaphore sync_sems[SYNC_SEMAPHORES];
    Mutex mutex;
    Semaphore *sync = nullptr;
+   std::atomic<bool> flushing = false;

    template <class T>
    T *allocate() {
+       if (flushing.load()) {
+           ERR_PRINT("aaaa!");
+       }
        // alloc size is size+T+safeguard
        uint32_t alloc_size = ((sizeof(T) + 8 - 1) & ~(8 - 1));
        uint64_t size = command_mem.size();
@@ -361,6 +365,10 @@ class CommandQueueMT {

    void _flush() {
        lock();
+       if (flushing.load()) {
+           return;
+       }
+       flushing.store(true);

        uint64_t read_ptr = 0;
        uint64_t limit = command_mem.size();
@@ -378,6 +386,7 @@ class CommandQueueMT {
        }

        command_mem.clear();
+       flushing.store(false);
        unlock();
    }

[unfa]

Is this a crash or a freeze? I've had a freeze on project open that required xkill, so likely an infinite loop somewhere in Godot's code

I've had Godot freeze as well requiring me to -SIGKILL it, though in this case it seems an endless loop causes something to fill up and trigger a full-on crash.

[unfa]

@bruvzg I've applied your patch and tested my project with it - it no longer crashes, but...

I am not sure what happened as the file explorer in Godot doesn't show my projects res:// contents.

I can use the Scene > Open... menu item to open my project's main scene and laod it - then Godot freezes.

btop shows the Godot editor process as sleeping.

I can interrupt it in gdb and get a backtrace, though the process did not crash - it's just frozen.

Thread 1 "godot.linuxbsd." received signal SIGINT, Interrupt.
0x00007ffff7b2e580 in ?? () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007ffff7b2e580 in ?? () from /usr/lib/libc.so.6
#1  0x00007ffff7b34afa in pthread_mutex_lock () from /usr/lib/libc.so.6
#2  0x0000555557a6776a in __gthread_mutex_lock (__mutex=0x555560546510) at /usr/include/c++/12.1.1/x86_64-pc-linux-gnu/bits/gthr-default.h:749
#3  0x0000555557a677ba in __gthread_recursive_mutex_lock (__mutex=0x555560546510) at /usr/include/c++/12.1.1/x86_64-pc-linux-gnu/bits/gthr-default.h:811
#4  0x0000555557a677ee in std::recursive_mutex::lock (this=0x555560546510) at /usr/include/c++/12.1.1/mutex:108
#5  0x0000555557f9c643 in MutexImpl<std::recursive_mutex>::lock (this=0x555560546510) at ./core/os/mutex.h:47
#6  MutexLock<MutexImpl<std::recursive_mutex> >::MutexLock (p_mutex=..., this=<optimized out>) at ./core/os/mutex.h:66
#7  GDScriptCache::get_full_script (p_path=..., r_error=@0x7fffffffa854: OK, p_owner=...) at modules/gdscript/gdscript_cache.cpp:189
#8  0x0000555557f44717 in ResourceFormatLoaderGDScript::load (this=0x55556054bb40, p_path=..., p_original_path=..., r_error=0x5555759c5904, p_use_sub_threads=false, r_progress=0x5555759c58f8, 
    p_cache_mode=ResourceFormatLoader::CACHE_MODE_REUSE) at modules/gdscript/gdscript.cpp:2348
#9  0x000055555bcd1ac7 in ResourceLoader::_load (p_path=..., p_original_path=..., p_type_hint=..., p_cache_mode=ResourceFormatLoader::CACHE_MODE_REUSE, r_error=0x5555759c5904, p_use_sub_threads=false, r_progress=0x5555759c58f8)
    at core/io/resource_loader.cpp:213
#10 0x000055555bcd21ab in ResourceLoader::_thread_load_function (p_userdata=0x5555759c58c8) at core/io/resource_loader.cpp:240
#11 0x000055555bcd3b57 in ResourceLoader::load (p_path=..., p_type_hint=..., p_cache_mode=ResourceFormatLoader::CACHE_MODE_REUSE, r_error=0x0) at core/io/resource_loader.cpp:568
#12 0x000055555acb2eb4 in ResourceLoaderText::load (this=0x7fffffffadd0) at scene/resources/resource_format_text.cpp:480
#13 0x000055555acc13f9 in ResourceFormatLoaderText::load (this=0x55556107b990, p_path=..., p_original_path=..., r_error=0x555579427714, p_use_sub_threads=false, r_progress=0x555579427708, 
    p_cache_mode=ResourceFormatLoader::CACHE_MODE_REUSE) at scene/resources/resource_format_text.cpp:1568
#14 0x000055555bcd1ac7 in ResourceLoader::_load (p_path=..., p_original_path=..., p_type_hint=..., p_cache_mode=ResourceFormatLoader::CACHE_MODE_REUSE, r_error=0x555579427714, p_use_sub_threads=false, r_progress=0x555579427708)
    at core/io/resource_loader.cpp:213
#15 0x000055555bcd21ab in ResourceLoader::_thread_load_function (p_userdata=0x5555794276d8) at core/io/resource_loader.cpp:240
#16 0x000055555bcd3b57 in ResourceLoader::load (p_path=..., p_type_hint=..., p_cache_mode=ResourceFormatLoader::CACHE_MODE_REUSE, r_error=0x0) at core/io/resource_loader.cpp:568
#17 0x000055555acb2eb4 in ResourceLoaderText::load (this=0x7fffffffb4e0) at scene/resources/resource_format_text.cpp:480
#18 0x000055555acc13f9 in ResourceFormatLoaderText::load (this=0x55556107b990, p_path=..., p_original_path=..., r_error=0x555579658aa4, p_use_sub_threads=false, r_progress=0x555579658a98, 
    p_cache_mode=ResourceFormatLoader::CACHE_MODE_REPLACE) at scene/resources/resource_format_text.cpp:1568
#19 0x000055555bcd1ac7 in ResourceLoader::_load (p_path=..., p_original_path=..., p_type_hint=..., p_cache_mode=ResourceFormatLoader::CACHE_MODE_REPLACE, r_error=0x555579658aa4, p_use_sub_threads=false, r_progress=0x555579658a98)
    at core/io/resource_loader.cpp:213
#20 0x000055555bcd21ab in ResourceLoader::_thread_load_function (p_userdata=0x555579658a68) at core/io/resource_loader.cpp:240
#21 0x000055555bcd3b57 in ResourceLoader::load (p_path=..., p_type_hint=..., p_cache_mode=ResourceFormatLoader::CACHE_MODE_REPLACE, r_error=0x7fffffffba40) at core/io/resource_loader.cpp:568
#22 0x0000555558fbac44 in EditorNode::load_scene (this=0x555562fe8560, p_scene=..., p_ignore_broken_deps=false, p_set_inherited=false, p_clear_errors=true, p_force_open_imported=false, p_silent_change_tab=false)
    at editor/editor_node.cpp:3718
#23 0x0000555558fa6e26 in EditorNode::_dialog_action (this=0x555562fe8560, p_file=...) at editor/editor_node.cpp:1890
#24 0x00005555590236a0 in call_with_variant_args_helper<EditorNode, String, 0ul> (p_instance=0x555562fe8560, p_method=(void (EditorNode::*)(EditorNode * const, String)) 0x555558fa6d12 <EditorNode::_dialog_action(String)>, 
    p_args=0x7fffffffc0c0, r_error=...) at ./core/variant/binder_common.h:262
#25 0x0000555559021e50 in call_with_variant_args<EditorNode, String> (p_instance=0x555562fe8560, p_method=(void (EditorNode::*)(EditorNode * const, String)) 0x555558fa6d12 <EditorNode::_dialog_action(String)>, p_args=0x7fffffffc0c0, 
    p_argcount=1, r_error=...) at ./core/variant/binder_common.h:376
#26 0x000055555901bfa6 in CallableCustomMethodPointer<EditorNode, String>::call (this=0x55556b72b740, p_arguments=0x7fffffffc0c0, p_argcount=1, r_return_value=..., r_call_error=...) at ./core/object/callable_method_pointer.h:104
#27 0x000055555bd8fa71 in Callable::callp (this=0x55556b475d68, p_arguments=0x7fffffffc0c0, p_argcount=1, r_return_value=..., r_call_error=...) at core/variant/callable.cpp:50
#28 0x000055555c0d529f in Object::emit_signalp (this=0x55556b320a30, p_name=..., p_args=0x7fffffffc0c0, p_argcount=1) at core/object/object.cpp:990
#29 0x00005555585e42e6 in Object::emit_signal<String> (this=0x55556b320a30, p_name=...) at ./core/object/object.h:818
#30 0x0000555558ea5126 in EditorFileDialog::_action_pressed (this=0x55556b320a30) at editor/editor_file_dialog.cpp:403
#31 0x0000555558ec85a1 in call_with_variant_args_helper<EditorFileDialog>(EditorFileDialog*, void (EditorFileDialog::*)(), Variant const**, Callable::CallError&, IndexSequence<>) (p_instance=0x55556b320a30, 
    p_method=(void (EditorFileDialog::*)(EditorFileDialog * const)) 0x555558ea4d9e <EditorFileDialog::_action_pressed()>, p_args=0x0, r_error=...) at ./core/variant/binder_common.h:262
#32 0x0000555558ec6f7b in call_with_variant_args<EditorFileDialog> (p_instance=0x55556b320a30, p_method=(void (EditorFileDialog::*)(EditorFileDialog * const)) 0x555558ea4d9e <EditorFileDialog::_action_pressed()>, p_args=0x0, 
    p_argcount=0, r_error=...) at ./core/variant/binder_common.h:376
#33 0x0000555558ec559c in CallableCustomMethodPointer<EditorFileDialog>::call (this=0x55556b3be810, p_arguments=0x0, p_argcount=0, r_return_value=..., r_call_error=...) at ./core/object/callable_method_pointer.h:104
#34 0x000055555bd8fa71 in Callable::callp (this=0x555572d73998, p_arguments=0x0, p_argcount=0, r_return_value=..., r_call_error=...) at core/variant/callable.cpp:50
#35 0x000055555c0d529f in Object::emit_signalp (this=0x55556b320a30, p_name=..., p_args=0x0, p_argcount=0) at core/object/object.cpp:990
#36 0x0000555557eb4b10 in Object::emit_signal<>(StringName const&) (this=0x55556b320a30, p_name=...) at ./core/object/object.h:818
#37 0x000055555a21fba6 in AcceptDialog::_ok_pressed (this=0x55556b320a30) at scene/gui/dialogs.cpp:107
#38 0x000055555a229a8a in call_with_variant_args_helper<AcceptDialog>(AcceptDialog*, void (AcceptDialog::*)(), Variant const**, Callable::CallError&, IndexSequence<>) (p_instance=0x55556b320a30, 
    p_method=(void (AcceptDialog::*)(AcceptDialog * const)) 0x55555a21fb36 <AcceptDialog::_ok_pressed()>, p_args=0x0, r_error=...) at ./core/variant/binder_common.h:262

I'll try some more things to see if this isn't my fault, though it's still awkward...

[unfa]

I've done more testing. All Godot 4 official alpha releases between 1 and 14 segfault on loading my project.

Also - the patch makes Godot crash consistently when I try to load any scene. I am not sure it helps any - the crashing is delayed, but it's not any more useful unfortunately.

[unfa]

I have this problem just resurfaced in another branch of my project:

Thread 1 "godot" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x000055555b57448f in CommandQueueMT::_flush (this=0x55556154e3d0) at ./core/templates/command_queue_mt.h:373
#2  0x000055555b574524 in CommandQueueMT::flush_if_pending (this=0x55556154e3d0) at ./core/templates/command_queue_mt.h:404
#3  0x000055555b58c714 in RenderingServerDefault::global_shader_parameter_get_type (this=0x55556154e190, p1=...) at servers/rendering/rendering_server_default.h:927
#4  0x000055555b5d2b84 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1352
#5  0x000055555b6053f6 in ShaderLanguage::_parse_shader (this=0x555561cbf158, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8355
#6  0x000055555b612309 in ShaderLanguage::compile (this=0x555561cbf158, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9750
#7  0x000055555b5d2c99 in ShaderCompiler::compile (this=0x555561cbf158, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffffffcd10, p_path=..., r_gen_code=...)
    at servers/rendering/shader_compiler.cpp:1363
#8  0x000055555b740f9b in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x555579c4ce80, p_code=...)
    at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#9  0x000055555b77e5b0 in RendererRD::MaterialStorage::shader_set_code (this=0x555561589b50, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2395
#10 0x000055555b5c4419 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc1cb878)
    at ./core/templates/command_queue_mt.h:322
#11 0x000055555b57448f in CommandQueueMT::_flush (this=0x55556154e3d0) at ./core/templates/command_queue_mt.h:373
#12 0x000055555b574524 in CommandQueueMT::flush_if_pending (this=0x55556154e3d0) at ./core/templates/command_queue_mt.h:404
#13 0x000055555b58a46f in RenderingServerDefault::canvas_item_clear (this=0x55556154e190, p1=...) at servers/rendering/rendering_server_default.h:862
#14 0x000055555a0b5293 in CanvasItem::_redraw_callback (this=0x5555669d6e60) at scene/main/canvas_item.cpp:130
#15 0x0000555558ef7f6a in call_with_variant_args_helper<CanvasItem>(CanvasItem*, void (CanvasItem::*)(), Variant const**, Callable::CallError&, IndexSequence<>) (p_instance=0x5555669d6e60, 
    p_method=(void (CanvasItem::*)(CanvasItem * const)) 0x55555a0b5238 <CanvasItem::_redraw_callback()>, p_args=0x0, r_error=...) at ./core/variant/binder_common.h:262
#16 0x0000555558ef5660 in call_with_variant_args<CanvasItem> (p_instance=0x5555669d6e60, p_method=(void (CanvasItem::*)(CanvasItem * const)) 0x55555a0b5238 <CanvasItem::_redraw_callback()>, p_args=0x0, 
    p_argcount=0, r_error=...) at ./core/variant/binder_common.h:376
#17 0x0000555558ef1ef6 in CallableCustomMethodPointer<CanvasItem>::call (this=0x5555767e04a0, p_arguments=0x0, p_argcount=0, r_return_value=..., r_call_error=...)
    at ./core/object/callable_method_pointer.h:104
#18 0x000055555bf759c9 in Callable::callp (this=0x7fffe6324ee8, p_arguments=0x0, p_argcount=0, r_return_value=..., r_call_error=...) at core/variant/callable.cpp:50
#19 0x000055555c2bc1e2 in MessageQueue::_call_function (this=0x55556056f4b0, p_callable=..., p_args=0x7fffe6324f00, p_argcount=0, p_show_error=false) at core/object/message_queue.cpp:229
#20 0x000055555c2bc531 in MessageQueue::flush (this=0x55556056f4b0) at core/object/message_queue.cpp:275
#21 0x000055555a15a8fa in SceneTree::physics_process (this=0x555563b9ed90, p_time=0.016666666666666666) at scene/main/scene_tree.cpp:429
#22 0x0000555557b96740 in Main::iteration () at main/main.cpp:2975
#23 0x0000555557b3c9f2 in OS_LinuxBSD::run (this=0x7fffffffd840) at platform/linuxbsd/os_linuxbsd.cpp:538
#24 0x0000555557b37250 in main (argc=4, argv=0x7fffffffdd78) at platform/linuxbsd/godot_linuxbsd.cpp:72

[reduz]

With 100% confidence, this specific bug (based on the stack trace posted here) was fixed by #66813 so it should not happen again. As such, I will close this issue. If for some reason you encounter it again its likely a different crash (as there is no way the stack dump is the same) so please open a new issue.

[unfa]

Thanks, I have not experienced this in Beta 11 so I think it is gone. Yay!