crash when starting godot under Linux4Tegra

[danboid]

Godot version:

3.1 release tarball

OS/device including version:

Ubuntu 18.04.2 Linux4Tegra running on Jetson Nano

Issue description:

godot crashes on startup with this error:

./godot.x11.tools.64 
handle_crash: Program crashed with signal 11
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] linux-vdso.so.1(__kernel_rt_sigreturn+0) [0x7f7ba876c0] (??:0)
[2] /usr/lib/aarch64-linux-gnu/libGLX.so.0(+0xd670) [0x7f7afb2670] (??:0)
-- END OF BACKTRACE --
handle_crash: Program crashed with signal 11
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] linux-vdso.so.1(__kernel_rt_sigreturn+0) [0x7f7ba876c0] (??:0)
[2] /usr/lib/aarch64-linux-gnu/libGLX.so.0(+0xd670) [0x7f7afb2670] (??:0)
-- END OF BACKTRACE --
handle_crash: Program crashed with signal 11
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] linux-vdso.so.1(__kernel_rt_sigreturn+0) [0x7f7ba876c0] (??:0)
[2] /usr/lib/aarch64-linux-gnu/libGLX.so.0(+0xd670) [0x7f7afb2670] (??:0)
-- END OF BACKTRACE --
Aborted (core dumped)

Note that I had to use qemu-user-static / systemd-nspawn on my laptop to build godot. L4T freezes part way through building godot on my Nano.

How much RAM is required to build godot under ARM64 Linux? 8GB seems to be enough as it builds on my laptop.

[danboid]

dan@dan-desktop:/usr/lib/aarch64-linux-gnu/tegra-egl$ ls -l total 1244 -rw-r--r-- 1 root root 37 Jan 28 2018 ld.so.conf -rwxr-xr-x 1 root root 1075704 Mar 13 10:03 libEGL_nvidia.so.0 -rwxr-xr-x 1 root root 67800 Mar 13 10:03 libGLESv1_CM_nvidia.so.1 -rwxr-xr-x 1 root root 116952 Mar 13 10:03 libGLESv2_nvidia.so.2 -rw-r--r-- 1 root root 107 Mar 13 10:03 nvidia.json dan@dan-desktop:/usr/lib/aarch64-linux-gnu/tegra-egl$

The GL include files are in the usual place, apart from that.

[nicoboss]

The exact same bug occurs when running Godot on the Nintendo Switch after booting Linux4Tegra. Because being able to directly test on the Nintendo Switch is important here a workaround until this bug is fixed. Just build it by following the X11 building guide and enter the following command to launch it:

LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgtk3-nocsd.so.0:/usr/lib/aarch64-linux-gnu/libXcursor.so.1:/usr/lib/aarch64-linux-gnu/libXinerama.so.1:/usr/lib/aarch64-linux-gnu/libXrandr.so.2:/usr/lib/aarch64-linux-gnu/libXrender.so.1:/usr/lib/aarch64-linux-gnu/libX11.so.6:/usr/lib/aarch64-linux-gnu/libXi.so.6:/usr/lib/aarch64-linux-gnu/libfreetype.so.6:/usr/lib/aarch64-linux-gnu/libpng16.so.16:/usr/lib/aarch64-linux-gnu/libasound.so.2:/usr/lib/aarch64-linux-gnu/libpulse.so.0:/lib/aarch64-linux-gnu/libz.so.1:/usr/lib/aarch64-linux-gnu/libGL.so.1:/lib/aarch64-linux-gnu/libpthread.so.0:/lib/aarch64-linux-gnu/libdl.so.2:/usr/lib/aarch64-linux-gnu/libstdc++.so.6:/lib/aarch64-linux-gnu/libm.so.6:/lib/aarch64-linux-gnu/libgcc_s.so.1:/lib/aarch64-linux-gnu/libc.so.6:/lib/ld-linux-aarch64.so.1:/usr/lib/aarch64-linux-gnu/libXfixes.so.3:/usr/lib/aarch64-linux-gnu/libXext.so.6:/usr/lib/aarch64-linux-gnu/libxcb.so.1:/usr/lib/aarch64-linux-gnu/pulseaudio/libpulsecommon-11.1.so:/lib/aarch64-linux-gnu/libdbus-1.so.3:/usr/lib/aarch64-linux-gnu/libGLX.so.0:/usr/lib/aarch64-linux-gnu/libGLdispatch.so.0:/usr/lib/aarch64-linux-gnu/libXau.so.6:/usr/lib/aarch64-linux-gnu/libXdmcp.so.6:/lib/aarch64-linux-gnu/libsystemd.so.0:/lib/aarch64-linux-gnu/libwrap.so.0:/usr/lib/aarch64-linux-gnu/libsndfile.so.1:/usr/lib/aarch64-linux-gnu/libasyncns.so.0:/lib/aarch64-linux-gnu/librt.so.1:/lib/aarch64-linux-gnu/libbsd.so.0:/lib/aarch64-linux-gnu/liblzma.so.5:/usr/lib/aarch64-linux-gnu/liblz4.so.1:/lib/aarch64-linux-gnu/libgcrypt.so.20:/lib/aarch64-linux-gnu/libnsl.so.1:/usr/lib/aarch64-linux-gnu/libFLAC.so.8:/usr/lib/aarch64-linux-gnu/libogg.so.0:/usr/lib/aarch64-linux-gnu/libvorbis.so.0:/usr/lib/aarch64-linux-gnu/libvorbisenc.so.2:/lib/aarch64-linux-gnu/libresolv.so.2:/lib/aarch64-linux-gnu/libgpg-error.so.0 ./godot.x11.tools.64

I tested the above workaround on latest master commit 2dc7be505a2b7bc803d3a3b2d6311d4a435a4234 on the Nintendo Switch using Linux4Tegra Ubuntu 18.04. In case the above workaround doesn’t work on newer versions because more libraries where added just add them to LD_PRELOAD at their intended position which you can see using ldd godot.x11.tools.64.

Why does this workaround work? I assume the problem is libGLX_nvidia not loading before/while loading libGLX and so libGLX not being able to get the GPU information. I tried preloading libGLX_nvidia using LD_PRELOAD=/usr/lib/aarch64-linux-gnu/tegra/libGLX_nvidia.so.0 ./godot.x11.tools.64 but that caused: Error initializing GLAD at drivers/gles3/rasterizer_gles3.cpp:141 and drivers/gles2/rasterizer_gles2.cpp:166 because of OpenGL not being able to initialize for some reason. I figured out that by using LD_PRELOAD libraries load properly so I just used ldd ./godot.x11.tools.64 and put them all in the same order and without any modifications inside LD_PRELOAD where they all getting preloaded properly before starting Godot and so overrides all wrongly loaded libraries. I have no idea why Godot doesn’t load them properly even though they are all there when checking ldd ./godot.x11.tools.64 but other projects had similar issues with Linux4Tegra like for example https://github.com/hrydgard/ppsspp/issues/12001 which solved it by hardcoding libGLX_nvidia inside their Linux4Tegra build configuration. Even though there’s now a workaround I really hope this issue is getting fixed soon because Linux4Tegra on the Switch got a quite popular Linux gaming configuration with a lot of developers not knowing this workaround also there are other platforms using Linux4Tegra like the Jetson Nano mentioned in the issue report. In the end it’s absolutely awesome the Godot Editor works on arm64 at all and especially with good enough performance to run at usable speed on the Nintendo Switch. Just write if you need any Linux4Tegra specific information or help testing potential fixes of this issue.

[danboid]

Thanks @nicoboss, I'll try your workaround soon, if one of the the devs doesn't fix it first.

[nicoboss]

I did some more testing and my assumption of libGLX_nvidia not loading seems right. When launching godot normally libGLX_nvidia doesn't get loaded at all while when preloading all the shared libraries libGLX_nvidia is the first detected dependency of loaded shared libraries that gets loader right after transferring the control of shared libraries to godot.x11.tools.64.

I figured out that my workaround only seems to work if every single required first degree dependent library will be preloaded. Even if only a single one is missing it won't be loaded properly by godot.x11.tools.64 and it crashes. You most likely have to write you own LD_PRELOAD command based on the output of ldd ./godot.x11.tools.64 if you don't have all the libraries at the exact same location or compiled godot with custom build configurations.

What’s also interesting is that setting LD_BIND_NOW = 1 will result in godot having the same error as when launched normally even though it’s quite similar to preloading everything using LD_PRELOAD.

Here the DEBUG_LD of normally loaded failing and LD_PRELOAD loaded working godot.x11.tools.64 startup log in order to help further investigation of this issue: godot_L4T_LD_DEBUG_normal_failing.log godot_L4T_LD_DEBUG_LD_PRELOAD_working.log

[KoBeWi]

Can anyone still reproduce this bug in Godot 3.2.3 or any later release?

[nicoboss]

@KoBeWi I just tested Godot 3.2.3 and latest master on latest Linux4Tegra on the Nintendo Switch (Switchroot Ubuntu 3.2.0 build with 18.04 LTS [Bionic Beaver] and a 4 GB swap file).

I was successfully able to compile and run Godot 3.2.3 in booth GLES2 and GLES3 mode and latest master https://github.com/godotengine/godot/commit/3dc8aaaccc642cddbd8d5c1841fef079db5c7edf in Vulkan mode without any issues.

This issue seems to be fixed and can be closed – thanks for your awesome work.