Inconsistent crashes when switching scenes

tralph3 commented 1 year ago

Godot version

4.0.3.stable

System information

Arch Linux - RX 570 - Ryzen 5 3400G - Vulkan

Issue description

When running the project, if I change scenes, for some reason the game sometimes crashes. It can work fine 20 times in a row, and then crash out of the blue, but it always happens when switching scenes, at no other time.

Here is the piece of code I use to change scenes with a transition:

extends CanvasLayer

func change_scene_to_file(target: String) -> void:
    $AnimationPlayer.play_backwards("fade")
    await $AnimationPlayer.animation_finished
    get_tree().change_scene_to_file(target)
    $AnimationPlayer.play("fade")

func reload_current_scene() -> void:
    $AnimationPlayer.play_backwards("fade")
    await $AnimationPlayer.animation_finished
    get_tree().reload_current_scene()
    $AnimationPlayer.play("fade")

The problem is not me sending an incorrect scene file path. I have corroborated that the path I am giving when it crashes is proper.

I launched godot through the terminal to see what it says when it crashes. Here's the result:

Godot Engine v4.0.3.stable.arch_linux - https://godotengine.org
Vulkan API 1.3.246 - Forward+ - Using Vulkan Device #0: AMD - AMD Radeon RX 570 Series (RADV POLARIS10)

ERROR: FATAL: Index p_index = 0 is out of bounds (count = 0).
   at: operator[] (./core/templates/local_vector.h:155)

================================================================
handle_crash: Program crashed with signal 4
Engine version: Godot Engine v4.0.3.stable.arch_linux
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[1] /usr/lib/libc.so.6(+0x39ab0) [0x7ff789562ab0] (??:0)
[2] /usr/bin/godot(+0x3da8c91) [0x5560e4399c91] (??:?)
[3] /usr/bin/godot(+0x3d09a4a) [0x5560e42faa4a] (??:?)
[4] /usr/bin/godot(+0x43da93c) [0x5560e49cb93c] (??:?)
[5] /usr/bin/godot(+0x4df1b3c) [0x5560e53e2b3c] (??:?)
[6] /usr/bin/godot(+0x4423665) [0x5560e4a14665] (??:?)
[7] /usr/bin/godot(+0x4a6fbc5) [0x5560e5060bc5] (??:?)
[8] /usr/bin/godot(+0x4425009) [0x5560e4a16009] (??:?)
[9] /usr/bin/godot(+0x420c1ec) [0x5560e47fd1ec] (??:?)
[10] /usr/bin/godot(+0x3d8c58b) [0x5560e437d58b] (??:?)
[11] /usr/bin/godot(+0x3d8c58b) [0x5560e437d58b] (??:?)
[12] /usr/bin/godot(+0x3d8c58b) [0x5560e437d58b] (??:?)
[13] /usr/bin/godot(+0x3d8c58b) [0x5560e437d58b] (??:?)
[14] /usr/bin/godot(+0x3de039a) [0x5560e43d139a] (??:?)
[15] /usr/bin/godot(+0x4230fee) [0x5560e4821fee] (??:?)
[16] /usr/bin/godot(+0x3de3db5) [0x5560e43d4db5] (??:?)
[17] /usr/bin/godot(+0x3338e86) [0x5560e3929e86] (??:?)
[18] /usr/bin/godot(+0x48ba7a8) [0x5560e4eab7a8] (??:?)
[19] /usr/bin/godot(+0x2e2d545) [0x5560e341e545] (??:?)
[20] /usr/bin/godot(+0x2e8a234) [0x5560e347b234] (??:?)
[21] /usr/bin/godot(+0x10fbd89) [0x5560e16ecd89] (??:?)
[22] /usr/bin/godot(+0x10a18b9) [0x5560e16928b9] (??:?)
[23] /usr/lib/libc.so.6(+0x23850) [0x7ff78954c850] (??:0)
[24] /usr/lib/libc.so.6(__libc_start_main+0x8a) [0x7ff78954c90a] (??:0)
[25] /usr/bin/godot(+0x10afda5) [0x5560e16a0da5] (??:?)
-- END OF BACKTRACE --
================================================================

Steps to reproduce

Change scenes with get_tree().change_scene_to_file()
With some "luck"... crash

Minimal reproduction project

N/A

akien-mga commented 1 year ago

Could you use the debug build from https://github.com/Calinou/godot-debug-builds/releases/tag/v4.0.3 to trigger this crash again and get a stacktrace with debug symbols?

Alternatively you can compile Godot with debug_symbols=yes. Or maybe Arch provides debuginfo packages since this seems to be a distro packaged version you're using.

tralph3 commented 1 year ago

Yes, here you go:

ERROR: FATAL: Index p_index = 0 is out of bounds (count = 0).
   at: operator[] (./core/templates/local_vector.h:155)

================================================================
handle_crash: Program crashed with signal 4
Engine version: Godot Engine v4.0.3.stable.custom_build (5222a99f5d38cd5346254cefed8f65315bca4fcb)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[1] /usr/lib/libc.so.6(+0x39ab0) [0x7f9c892aeab0] (??:0)
[2] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x43883f8) [0x555dd54b33f8] (/opt/godot/./core/templates/local_vector.h:155)
[3] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x43a8a9a) [0x555dd54d3a9a] (/opt/godot/./core/math/vector3.h:196)
[4] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x4c0b138) [0x555dd5d36138] (/opt/godot/servers/physics_3d/gjk_epa.cpp:135)
[5] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x4c0d901) [0x555dd5d38901] (/opt/godot/./core/math/vector3.h:196 (discriminator 4))
[6] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x4c09f18) [0x555dd5d34f18] (/opt/godot/servers/physics_3d/gjk_epa.cpp:899)
[7] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x4c0ac01) [0x555dd5d35c01] (/opt/godot/servers/physics_3d/gjk_epa.cpp:999)
[8] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x496f59c) [0x555dd5a9a59c] (/opt/godot/servers/physics_3d/godot_collision_solver_3d.cpp:440)
[9] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x438d82b) [0x555dd54b882b] (/opt/godot/servers/physics_3d/godot_shape_3d.cpp:1440)
[10] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x438d82b) [0x555dd54b882b] (/opt/godot/servers/physics_3d/godot_shape_3d.cpp:1440)
[11] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x438d82b) [0x555dd54b882b] (/opt/godot/servers/physics_3d/godot_shape_3d.cpp:1440)
[12] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x438d82b) [0x555dd54b882b] (/opt/godot/servers/physics_3d/godot_shape_3d.cpp:1440)
[13] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x438d94e) [0x555dd54b894e] (/opt/godot/servers/physics_3d/godot_shape_3d.h:449)
[14] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x49729fe) [0x555dd5a9d9fe] (/opt/godot/servers/physics_3d/godot_collision_solver_3d.cpp:579)
[15] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x43a3839) [0x555dd54ce839] (/opt/godot/servers/physics_3d/godot_space_3d.cpp:322)
[16] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x364e0c2) [0x555dd47790c2] (/opt/godot/./core/object/ref_counted.h:220)
[17] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x52c3898) [0x555dd63ee898] (/opt/godot/core/object/object.cpp:792)
[18] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x30046b2) [0x555dd412f6b2] (/opt/godot/scene/main/scene_tree.cpp:874)
[19] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0x300a561) [0x555dd4135561] (/opt/godot/scene/main/scene_tree.cpp:427)
[20] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0xbb6de7) [0x555dd1ce1de7] (/opt/godot/main/main.cpp:3131)
[21] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0xb51ba1) [0x555dd1c7cba1] (/opt/godot/platform/linuxbsd/os_linuxbsd.cpp:889)
[22] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0xb421f1) [0x555dd1c6d1f1] (/opt/godot/platform/linuxbsd/godot_linuxbsd.cpp:75)
[23] /usr/lib/libc.so.6(+0x23850) [0x7f9c89298850] (??:0)
[24] /usr/lib/libc.so.6(__libc_start_main+0x8a) [0x7f9c8929890a] (??:0)
[25] /home/tralph3/Downloads/godot-4.0.3-editor-debug-linux-gcc9.3/bin/linux/godot.linuxbsd.editor.x86_64(+0xb4f16e) [0x555dd1c7a16e] (??:?)
-- END OF BACKTRACE --
================================================================

akien-mga commented 1 year ago

This shows that it's crashing in the 3D physics collision solver, so that's helpful!

Check the line numbers where it's crashing in the 4.0.3-stable might help physics contributors guess what the issue could be. But of course the best to really identify the problem would be to have a clear way to reproduce it, though I know it's not necessarily easy to create a reproduction project for an intermittent crash.

tralph3 commented 1 year ago

There are physics forces being applied until the very last frame when changing scenes. Could that be causing a problem?

tralph3 commented 1 year ago

I've been investigating the issue, and I just can't find a reason for it. Maybe I could give someone here access to the repo, see if they can manage to debug it. Someone who has a better understanding of the inner workings of the engine.

I have found that going from level 1 to level 2, then restarting level 2 is a guaranteed crash. Going to level 3 is a guaranteed crash as well (but not directly loading the scene). Also, restarting level 3 is a guaranteed crash.

The issue is present on both Linux (AMD and Intel GPUs) and Windows (NVIDIA GPU). All of them using Vulkan. I didn't test other APIs, although I doubt it would make much difference.

I know this is meaningless without access to the project, so if someone is willing to dig deeper into this, please let me know.

tralph3 commented 1 year ago

I tried to debug this. Apparently, (although I'm not certain), the issue is caused by accessing nodes belonging to the scene that's being unloaded.

For some reason, when I restart a level that crashes, an Area3D that's present on the level sends a signal that a body entered, then it checks a property of that body. It is then that the crash occurs.

Oddly enough, if I remove this Area from the level, that piece of code doesn't execute of course, but an _input function I have on an autoloaded UI controller does, and it crashes there.

tralph3 commented 1 year ago

I'll leave the previous comment merely to provide more information, but on closer inspection, it seems that it was consistently crashing at the same line because of time, not because that line was actually doing something wrong. If I remove that piece of code, then it consistently crashes on some other line of code. If I remove that one, then on another one, and so on.

All these lines of code have no relevance to each other. Some are input functions, others are functions called from signals, one of them is even a variable defined when the scene is read (@onready var ...).

There's no pattern to follow here. I can't find a fault in my code at all. This must have something to do with the engine itself.

I looked through the engine code too, I can't really understand it since I don't know how it all works and is tied together. It seems to be trying to detect a collision tho, from what I can gather, which is odd.

Removing the player, which is the only rigidbody in the scene, does not solve the issue.

tralph3 commented 1 year ago

I figured it out. It's grid maps. The thing that was causing the crash was a grid map. If I remove it, it works no problems.

Problem is, the grid map is the level. Guess I will drop them. People have warned me about them being limited and outdated, guess they were right.

At least we know now where to look to fix this.

tralph3 commented 1 year ago

I'm trying to cobble together a test project to reproduce the issue. I don't know what exactly my project has that makes it be, but I'm not finding a way to do it.

It's weird too, since all my levels are composed of the same nodes. A GridMap, an Environment, the Player, a custom node for the player camera, a Goal which is just a grid map with a specific shape and some particle effects, and maybe some extra things for that particular level.

All the other levels may crash, but Level 3 is the only one that consistently crashes. It does have a difference with other levels, it contains meshes directly imported from blender. Although at some point, level 1 did have meshes (but those where gltf).

However, removing the meshes does not solve the issue. Only removing the gridmap resolves it.

tralph3 commented 1 year ago

Further developments. The GridMap in Level 3 was slightly moved. It's position was Vector3(0.008, 0.00, -0.02). After moving it to Vector3(0, 0, 0) the crash doesn't occur anymore.

I then moved it back to it's previous coordinates by pressing Ctrl+Z and the crash happened again, consistently when I restarted the level.

I then put it at 0,0,0 once more, and the crash fixed, and upon setting the Vector3(0.008, 0.00, -0.02) coordinates by hand the crash also didn't happen again.

I am now unable to reproduce the consistent crashing.

Copying these values to the test project in hopes to get a reproduceable crash didn't make it crash.

GridMaps are messy.

YuriSizov commented 1 year ago

Hey, don't let our lack of response discourage you. Things take time to get noticed sometimes. But you're doing a great job researching and identifying the issue, so thanks a lot!

bugbountyguy commented 10 hours ago

I just wanted to share that I was having the same stack trace and issue with trying to change scenes in one of my projects. I was using the Time.get_unix_time_from_system() shortly before changing scenes in some logic to prevent double clicks (for steam deck, legion go, etc. with touch being counted as both a mouse click and touch at the same time) and noticed it crashed here when changing scenes.

I then found I could get past this crash by adding an await timeout immediately before changing scenes: await get_tree().create_timer(0.00001).timeout and it never crashes after that. I think there may be some kind of race condition going on with some internal functions, perhaps when calling static methods (like those in Time) right before changing scenes, but I'm not sure. More interestingly, if I put a print statement right before the timeout and after, it only prints the one after the timeout (even if I put the timeout at 1.0 seconds instead).

For the reported ticket here, I see animation being used - perhaps Time is being used in that functionality somewhere as well to play the animation?

Hope this helps identify the issue from the Godot side. I'm on v4.3 and using the Linux (not .net) build. Interestingly it didn't crash on my laptop, also using the same version and build, but does crash on the mobile pc

EDIT: Okay well the original stack trace looked the same, but I guess the debug stack trace is a lil different. But still wondering if it's a race condition somewhere :) Here's my debugger's stack trace:

================================================================
handle_crash: Program crashed with signal 11
Engine version: Godot Engine v4.3.stable.custom_build (77dcf97d82cbfe4e4615475fa52ca03da645dbd8)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[1] /lib64/libc.so.6(+0x40d00) [0x7fe0f4a3ed00] (??:0)
[2] /var/home/user/Downloads/bin/linux/godot.linuxbsd.editor.x86_64(+0x31a21e5) [0x55ade58001e5] (/opt/godot/scene/3d/node_3d.cpp:345)
[3] /var/home/user/Downloads/bin/linux/godot.linuxbsd.editor.x86_64(+0x2b71dbd) [0x55ade51cfdbd] (/opt/godot/scene/main/viewport.cpp:4208)
[4] /var/home/user/Downloads/bin/linux/godot.linuxbsd.editor.x86_64(+0x2b740b3) [0x55ade51d20b3] (/opt/godot/scene/main/viewport.cpp:903)
[5] /var/home/user/Downloads/bin/linux/godot.linuxbsd.editor.x86_64(+0x7dff5b) [0x55ade2e3df5b] (/opt/godot/./core/variant/variant.h:801)
[6] /var/home/user/Downloads/bin/linux/godot.linuxbsd.editor.x86_64(+0x526036c) [0x55ade78be36c] (/opt/godot/core/object/object.cpp:808)
[7] /var/home/user/Downloads/bin/linux/godot.linuxbsd.editor.x86_64(+0x2b1719d) [0x55ade517519d] (/opt/godot/./core/variant/variant.h:308)
[8] /var/home/user/Downloads/bin/linux/godot.linuxbsd.editor.x86_64(+0x2b220b4) [0x55ade51800b4] (/opt/godot/./core/variant/variant.h:308)
[9] /var/home/user/Downloads/bin/linux/godot.linuxbsd.editor.x86_64(+0x66f17a) [0x55ade2ccd17a] (/opt/godot/main/main.cpp:4070)
[10] /var/home/user/Downloads/bin/linux/godot.linuxbsd.editor.x86_64(+0x602791) [0x55ade2c60791] (/opt/godot/platform/linuxbsd/os_linuxbsd.cpp:962)
[11] /var/home/user/Downloads/bin/linux/godot.linuxbsd.editor.x86_64(+0x5ce9fc) [0x55ade2c2c9fc] (/opt/godot/platform/linuxbsd/godot_linuxbsd.cpp:85)
[12] /lib64/libc.so.6(+0x2a088) [0x7fe0f4a28088] (??:0)
[13] /lib64/libc.so.6(__libc_start_main+0x8b) [0x7fe0f4a2814b] (??:0)
[14] /var/home/user/Downloads/bin/linux/godot.linuxbsd.editor.x86_64(+0x5ffcfe) [0x55ade2c5dcfe] (??:?)
-- END OF BACKTRACE --
================================================================

godotengine / godot