Godot 4.3 Crashes During Import of Large Asset Directory

WalkingGlitch commented 1 month ago

Tested versions

Reproducible in Godot Engine v4.3.stable.flathub.77dcf97d8. This also occurs on the 4.3 Steam release as well. This did not occur in the 4.2.1 release.

EDIT: It does occur in the 4.2.2 release on Steam, but instead of crashing, it becomes unresponsive while maxing my CPU cores. I let it run for 25 minutes, and it was still unresponsive. This is on an AMD 5800X3D, so it's unlikely that I killed it too soon.

System information

MX Linux 23.3 6.9.12-2-liquorix-amd64

Issue description

Importing a large directory of assets causes a crash when the editor attempts to import and index them. To give an idea: 79926 items (76209 files, 3716 folders) 643.0 MiB (674,255,333 bytes)

I get the following output from the terminal running it with the --verbose flag:

Owner@mx:~
$ /usr/bin/flatpak run --branch=stable --arch=x86_64 --verbose --command=godot --file-forwarding org.godotengine.Godot @@ %f @@
F: No installations directory in /etc/flatpak/installations.d. Skipping
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/Owner/.local/share/flatpak
F: Opening user flatpak installation at path /home/Owner/.local/share/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: Skipping parental controls check for app/org.godotengine.Godot/x86_64/stable since parental controls are disabled globally
F: Opening user flatpak installation at path /home/Owner/.local/share/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: /var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/23.08/be4a045f86be2b8a7a592bab299c7dd41c174eba94ab8048401b1fa01c9eb86a/files/lib32 does not exist
F: Cleaning up unused container id 2813547324
F: Cleaning up per-app-ID state for org.godotengine.Godot
F: Allocated instance id 3955385820
F: Add defaults in dir /org/godotengine/Godot/
F: Add locks in dir /org/godotengine/Godot/
F: Allowing host-fs access
F: Not sharing "/run/media" with sandbox: Unable to open path "/run/media": No such file or directory
F: Allowing wayland access
F: Allowing x11 access
F: Allowing pulseaudio access
F: Pulseaudio user configuration file '/home/Owner/.config/pulse/client.conf': Error opening file /home/Owner/.config/pulse/client.conf: No such file or directory
F: Failed to run in transient scope: No systemd user session available, cgroups not available
F: Running 'bwrap --args 40 -- xdg-dbus-proxy --args=42'
F: Running 'bwrap --args 40 -- godot '%f''
Godot Engine v4.3.stable.flathub.77dcf97d8 - https://godotengine.org
Inconsistent value (1) for DRI_PRIME. Should be < 1 (GPU devices count). Using: 0
OpenGL API 4.6 (Core Profile) Mesa 24.1.3 (git-0c49f54c76) - Compatibility - Using Device: AMD - AMD Radeon RX 6900 XT (radeonsi, navi21, LLVM 17.0.6, DRM 3.57, 6.9.12-2-liquorix-amd64)

Editing project: /home/Owner/Godot/mastermind-test
Godot Engine v4.3.stable.flathub.77dcf97d8 - https://godotengine.org
Vulkan 1.3.278 - Forward Mobile - Using Device #0: AMD - AMD Radeon RX 6900 XT (RADV NAVI21)

Owner@mx:~
$ ERROR: Condition "line.size() <= 1" is true. Returning: ERR_PARSE_ERROR
   at: import (editor/import/resource_importer_csv_translation.cpp:95)
ERROR: Error importing 'res://MasterMind Assets/symbolClass/symbols.csv'.
   at: _reimport_file (editor/editor_file_system.cpp:2607)
ERROR: Caller thread can't call this function in this node (/root). Use call_deferred() or call_thread_group() instead.
   at: propagate_notification (scene/main/node.cpp:2422)

================================================================
handle_crash: Program crashed with signal 11
Engine version: Godot Engine v4.3.stable.flathub (77dcf97d82cbfe4e4615475fa52ca03da645dbd8)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[1] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x3ee80) [0x7fd0ab550e80] (??:0)
[2] /app/bin/godot-bin(+0x132f1e3) [0x5b7e28b311e3] (??:?)
[3] /app/bin/godot-bin(+0x132f2d7) [0x5b7e28b312d7] (??:?)
[4] /app/bin/godot-bin(+0x132f569) [0x5b7e28b31569] (??:?)
[5] /app/bin/godot-bin(+0x132facb) [0x5b7e28b31acb] (??:?)
[6] /app/bin/godot-bin(+0x132fcde) [0x5b7e28b31cde] (??:?)
[7] /app/bin/godot-bin(+0x13300e9) [0x5b7e28b320e9] (??:?)
[8] /app/bin/godot-bin(+0x1331ae7) [0x5b7e28b33ae7] (??:?)
[9] /app/bin/godot-bin(+0x132ec77) [0x5b7e28b30c77] (??:?)
[10] /app/bin/godot-bin(+0x132e068) [0x5b7e28b30068] (??:?)
[11] /app/bin/godot-bin(+0x1348efe) [0x5b7e28b4aefe] (??:?)
[12] /app/bin/godot-bin(+0x134ac14) [0x5b7e28b4cc14] (??:?)
[13] /app/bin/godot-bin(+0x1348c72) [0x5b7e28b4ac72] (??:?)
[14] /app/bin/godot-bin(+0x134ac14) [0x5b7e28b4cc14] (??:?)
[15] /app/bin/godot-bin(+0x1348c72) [0x5b7e28b4ac72] (??:?)
[16] /app/bin/godot-bin(+0x134ac14) [0x5b7e28b4cc14] (??:?)
[17] /app/bin/godot-bin(+0x1348c72) [0x5b7e28b4ac72] (??:?)
[18] /app/bin/godot-bin(+0x134ac14) [0x5b7e28b4cc14] (??:?)
[19] /app/bin/godot-bin(+0x1348c72) [0x5b7e28b4ac72] (??:?)
[20] /app/bin/godot-bin(+0x1348f86) [0x5b7e28b4af86] (??:?)
[21] /app/bin/godot-bin(+0x134706e) [0x5b7e28b4906e] (??:?)
[22] /app/bin/godot-bin(+0x1336046) [0x5b7e28b38046] (??:?)
[23] /app/bin/godot-bin(+0x133693b) [0x5b7e28b3893b] (??:?)
[24] /app/bin/godot-bin(+0x1336a30) [0x5b7e28b38a30] (??:?)
[25] /app/bin/godot-bin(+0x1336e0d) [0x5b7e28b38e0d] (??:?)
[26] /app/bin/godot-bin(+0x48d1d17) [0x5b7e2c0d3d17] (??:?)
[27] /app/bin/godot-bin(+0x1cfa667) [0x5b7e294fc667] (??:?)
[28] /app/bin/godot-bin(+0x185d695) [0x5b7e2905f695] (??:?)
[29] /app/bin/godot-bin(+0x185fbaa) [0x5b7e29061baa] (??:?)
[30] /app/bin/godot-bin(+0x4d5dbc3) [0x5b7e2c55fbc3] (??:?)
[31] /app/bin/godot-bin(+0x4d5e46c) [0x5b7e2c56046c] (??:?)
[32] /app/bin/godot-bin(+0x4794add) [0x5b7e2bf96add] (??:?)
[33] /app/bin/godot-bin(+0x4faaeb4) [0x5b7e2c7aceb4] (??:?)
[34] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x8ee39) [0x7fd0ab5a0e39] (??:0)
[35] /usr/lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fd0ab6289c4] (??:0)
-- END OF BACKTRACE --
================================================================

It also keeps the focus of my terminal emulator despite the process having completely crashed and no longer having a PID. I have to CTRL+C to escape it.

Steps to reproduce

This is a real simple one. Create a new project with a new directory.
Copy the assets to the root of the directory.
Select the //res folder to initiate the import.
Wait for the crash to happen.

I have put a download of a 7zip archive of the asset directory that causes the crash in the MRP.

Minimal reproduction project (MRP)

MRP.zip

Asset directory: https://mega.nz/file/eyhBkBAK#mDAghej8jNCh62ZaKzWqSq0xJ8flDerRUHCUPDx4zUY

Please only download this if you are actually working on the troubleshooting, I have limited bandwidth allotment for this.

cryoboltinteractive commented 1 week ago

Can you confirm the memory usage during import? We are facing a similar issue at #97610

Rubonnek commented 1 week ago

I was able to reproduce the issue in Arch Linux with a build at commit 06d51891678e3abf360d6fcd2e8bd1ba96704fcc.

I was also able to import all the assets successfully after setting Editor/Import/Use Multiple Threads to false in the Project Settings.

There may be a race condition somewhere in the import process.

MDIVS commented 6 days ago

Getting the same error.

Adding

[editor]

import/use_multiple_threads=false

on project.godot did not work for me.

RAM memory usage below 800 MB while importing assets.

Windows 11

CPU

AMD Ryzen 5 1600 Six-Core Processor 

Base speed: 3.20 GHz
Sockets:    1
Cores:  6
Logical processors: 12
Virtualisation: Enabled
L1 cache:   576 KB
L2 cache:   3.0 MB
L3 cache:   16.0 MB

Memory

32.0 GB

Speed:  2666 MT/s
Slots used: 2 of 4

Disk 1 (C:)

WALRAM 1TB

Capacity:   932 GB
Formatted:  932 GB
System disk:    Yes
Page file:  Yes
Type:   SSD

Read speed  32.8 KB/s
Write speed 4.1 KB/s
Active time 0%
Average response time   0.6 ms

GPU 0

AMD Radeon RX 580 2048SP

Driver version: 31.0.21018.5001
Driver date:    25/07/2023
DirectX version:    12 (FL 12.0)

godotengine / godot