Some unit tests fail on Debian bookworm and unstable

twrightsman commented 3 days ago

Tested versions

Reproducible in 4.3-stable

System information

Godot v4.3.stable (17df8f24f) - Debian GNU/Linux 12 (bookworm) 12 - Wayland - GLES3 (Compatibility) - Mesa Intel(R) UHD Graphics 620 (KBL GT2) - Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz (8 Threads)

Issue description

I am working to package 4.3-stable in Debian but have encountered a few unit test failures on both Debian stable/bookworm as well as bleeding-edge unstable (in a VM). Apologies if this is too many issues at once. I am happy to split this up further if that's easier! The logs are mostly identical, so here is the result from inside the unstable VM:

SCRIPT ERROR: Trying to assign value of type 'AnimationPlayer' to a variable of type 'Area2D'.
          at: @implicit_new (res://completion/get_node/member_typehint_scene_incompatible/native_member_typehint_scene_incompatible.gd:3)
SCRIPT ERROR: Trying to assign value of type 'Node' to a variable of type 'Area2D'.
          at: @implicit_new (res://completion/get_node/member_typehint_scene_incompatible/class_member_typehint_scene_incompatible.gd:3)
ERROR: Cannot open CryptoKeyMbedTLS file '/usr/bin/../tests/data/crypto/in.key'.
   at: load (modules/mbedtls/crypto_mbedtls.cpp:61)
ERROR: Cannot open CryptoKeyMbedTLS file '/usr/bin/../tests/data/crypto/in.pub'.
   at: load (modules/mbedtls/crypto_mbedtls.cpp:61)
ERROR: Cannot open CryptoKeyMbedTLS file '/usr/bin/../tests/data/crypto/in.key'.
   at: load (modules/mbedtls/crypto_mbedtls.cpp:61)
ERROR: Cannot save CryptoKeyMbedTLS file '/usr/bin/../tests/data/crypto/out.key'.
   at: save (modules/mbedtls/crypto_mbedtls.cpp:84)
[doctest] doctest version is "2.4.11"
[doctest] run with "--help" for options
===============================================================================
./modules/gdscript/tests/gdscript_test_runner_suite.h:43:
TEST SUITE: [Modules][GDScript]
TEST CASE:  Script compilation and runtime

modules/gdscript/tests/gdscript_test_runner.cpp:217: ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: /mnt/godot-build/godot3/modules/gdscript/tests/scripts/parser/warnings/confusable_identifier.gd
          GDTEST_OK
0 1

modules/gdscript/tests/gdscript_test_runner.cpp:217: ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: /mnt/godot-build/godot3/modules/gdscript/tests/scripts/parser/errors/identifier_similar_to_keyword.gd
          GDTEST_OK
>> WARNING
>> Line: 3
>> UNASSIGNED_VARIABLE
>> The variable "аs" was used before being assigned a value.
<null>

modules/gdscript/tests/gdscript_test_runner.cpp:217: ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: /mnt/godot-build/godot3/modules/gdscript/tests/scripts/analyzer/features/warning_ignore_targets.gd
          GDTEST_OK
>> WARNING
>> Line: 8
>> NARROWING_CONVERSION
>> Narrowing conversion (float is converted to int and loses precision).
>> WARNING
>> Line: 19
>> NARROWING_CONVERSION
>> Narrowing conversion (float is converted to int and loses precision).
>> WARNING
>> Line: 24
>> NARROWING_CONVERSION
>> Narrowing conversion (float is converted to int and loses precision).
>> WARNING
>> Line: 31
>> NARROWING_CONVERSION
>> Narrowing conversion (float is converted to int and loses precision).
>> WARNING
>> Line: 35
>> NARROWING_CONVERSION
>> Narrowing conversion (float is converted to int and loses precision).

./modules/gdscript/tests/gdscript_test_runner_suite.h:49: FATAL ERROR: REQUIRE( fail_count == 0 ) is NOT correct!
  values: REQUIRE( 3 == 0 )
  logged: Make sure `*.out` files have expected results.
          All GDScript tests should pass.

===============================================================================
./modules/mbedtls/tests/test_crypto_mbedtls.h:64:
TEST CASE:  [Crypto] CryptoKey is_public_only

modules/mbedtls/tests/test_crypto_mbedtls.cpp:86: ERROR: CHECK( is_equal ) is NOT correct!
  values: CHECK( false )

===============================================================================
./modules/mbedtls/tests/test_crypto_mbedtls.h:71:
TEST CASE:  [Crypto] CryptoKey save

modules/mbedtls/tests/test_crypto_mbedtls.cpp:73: FATAL ERROR: REQUIRE( file_access.is_valid() ) is NOT correct!
  values: REQUIRE( false )
./modules/mbedtls/tests/test_crypto_mbedtls.h:71: FATAL ERROR: test case CRASHED: SIGSEGV - Segmentation violation signal

===============================================================================
[doctest] test cases:    16 |    13 passed | 3 failed | 946 skipped
[doctest] assertions: 56651 | 56645 passed | 6 failed |
[doctest] Status: FAILURE!

================================================================
handle_crash: Program crashed with signal 11
Engine version: Godot Engine v4.3.stable.debian4.3-stable-1 (bf909ee9a75df1972cccdf4ad08d4f4820b14b58)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
ERROR: FATAL: Index p_index = 1 is out of bounds (size() = 0).
   at: get (./core/templates/cowdata.h:205)

Compiling on bookworm and running the tests does successfully produce a backtrace:

Dumping the backtrace. Please include this when reporting the bug to the project developer.
[1] /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x7fa7b2c5b050] (??:0)
[2] godot4(+0x141dce6) [0x560b018a9ce6] (??:?)
[3] godot4(+0x141e3cb) [0x560b018aa3cb] (??:?)
[4] godot4(+0x77d339) [0x560b00c09339] (??:0)
[5] godot4(+0xcee450) [0x560b0117a450] (??:0)
[6] godot4(+0x78a91b) [0x560b00c1691b] (??:0)
[7] godot4(+0x6b6c58) [0x560b00b42c58] (??:0)
[8] godot4(+0x5e6a53) [0x560b00a72a53] (??:0)
[9] /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7fa7b2c4624a] (??:0)
[10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7fa7b2c46305] (??:0)
[11] godot4(+0x6221e1) [0x560b00aae1e1] (??:0)
-- END OF BACKTRACE --
================================================================

On the unstable VM there is also Illegal instruction reported at the very end and on stable it instead shows Aborted.

Steps to reproduce

For Debian stable/bookworm:

sudo apt install build-essential ca-certificates libasound2-dev libenet-dev libfreetype-dev libgl-dev libglu1-mesa-dev libpcre2-dev libpng-dev libpulse-dev libsquish-dev libmbedtls-dev libtheora-dev libudev-dev libvorbis-dev libwebp-dev libx11-dev libxcursor-dev libxi-dev libxinerama-dev libxrandr-dev pkgconf scons zlib1g-dev libwayland-dev libgraphite2-dev libharfbuzz-dev libogg-dev libzstd-dev libicu-dev libbrotli-dev
scons target=editor arch=x86_64 platform=linuxbsd system_certs_path=/etc/ssl/certs/ca-certificates.crt tests=yes builtin_brotli=no builtin_certs=no builtin_embree=yes builtin_enet=no builtin_freetype=no builtin_graphite=no builtin_harfbuzz=no builtin_icu4c=no builtin_libogg=no builtin_libpng=no builtin_libtheora=no builtin_libvorbis=no builtin_libwebp=no builtin_mbedtls=no builtin_pcre2=no builtin_squish=no builtin_zlib=no builtin_zstd=no openxr=no use_static_cpp=no engine_update_check=no -j 8 CCFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security" LINKFLAGS="-Wl,-z,relro -Wl,-z,now" CFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security"
./bin/godot.linuxbsd.editor.x86_64 --test

For unstable I just add libembree-dev and use builtin_embree=no, as unstable has a recent enough libembree.

Minimal reproduction project (MRP)

N/A

akien-mga commented 2 days ago

The error look as if you'd be running the tests from a Godot binary in /usr/bin, but they're only supported when ran "in place" in the Git repository.

ERROR: Cannot open CryptoKeyMbedTLS file '/usr/bin/../tests/data/crypto/in.key'.

That path should be /mnt/godot-build/godot3/bin/../tests/data/crypto/in.key.

I know distro packagers tend to patch software to match distro requirements (I've been there myself on Mageia/Fedora), do you have any downstream patches that could impact this? Or are you maybe running the tests from the binary already installed in /usr/bin or PATH inadvertently?

If you can share the debian/ files, I'm familiar with the system and might be able to spot what's wrong.

Compiling on bookworm and running the tests does successfully produce a backtrace:

To get a proper backtrace you should compile with debug_symbols=yes passed explicitly.

twrightsman commented 2 days ago

Sorry for the confusion! I did indeed run the Godot binaries from /usr/bin instead of the command I wrote.

Copying the binaries into bin/ and running them on Debian unstable gives only a few failing tests for the editor (the release template passed all tests :tada:):

Editor Test Output

``` SCRIPT ERROR: Trying to assign value of type 'AnimationPlayer' to a variable of type 'Area2D'. at: @implicit_new (res://completion/get_node/member_typehint_scene_incompatible/native_member_typehint_scene_ incompatible.gd:3) SCRIPT ERROR: Trying to assign value of type 'Node' to a variable of type 'Area2D'. at: @implicit_new (res://completion/get_node/member_typehint_scene_incompatible/class_member_typehint_scene_i ncompatible.gd:3) WARNING: ICU data is not loaded, Unicode security and spoofing detection disabled. at: _is_valid_identifier (modules/text_server_adv/text_server_adv.cpp:7197) [doctest] doctest version is "2.4.11" [doctest] run with "--help" for options =============================================================================== ./modules/gdscript/tests/gdscript_test_runner_suite.h:43: TEST SUITE: [Modules][GDScript] TEST CASE: Script compilation and runtime modules/gdscript/tests/gdscript_test_runner.cpp:217: ERROR: CHECK( result.passed ) is NOT correct! values: CHECK( false ) logged: /mnt/godot-build/godot3/modules/gdscript/tests/scripts/parser/warnings/confusable_identifier.gd GDTEST_OK 0 1 modules/gdscript/tests/gdscript_test_runner.cpp:217: ERROR: CHECK( result.passed ) is NOT correct! values: CHECK( false ) logged: /mnt/godot-build/godot3/modules/gdscript/tests/scripts/parser/errors/identifier_similar_to_keyword.gd GDTEST_OK >> WARNING >> Line: 3 >> UNASSIGNED_VARIABLE >> The variable "аs" was used before being assigned a value. modules/gdscript/tests/gdscript_test_runner.cpp:217: ERROR: CHECK( result.passed ) is NOT correct! values: CHECK( false ) logged: /mnt/godot-build/godot3/modules/gdscript/tests/scripts/analyzer/features/warning_ignore_targets.gd GDTEST_OK >> WARNING >> Line: 8 >> NARROWING_CONVERSION >> Narrowing conversion (float is converted to int and loses precision). >> WARNING >> Line: 19 >> NARROWING_CONVERSION >> Narrowing conversion (float is converted to int and loses precision). >> WARNING >> Line: 24 >> NARROWING_CONVERSION >> Narrowing conversion (float is converted to int and loses precision). >> WARNING >> Line: 31 >> NARROWING_CONVERSION >> Narrowing conversion (float is converted to int and loses precision). >> WARNING >> Line: 35 >> NARROWING_CONVERSION >> Narrowing conversion (float is converted to int and loses precision). ./modules/gdscript/tests/gdscript_test_runner_suite.h:49: FATAL ERROR: REQUIRE( fail_count == 0 ) is NOT correct! values: REQUIRE( 3 == 0 ) logged: Make sure `*.out` files have expected results. All GDScript tests should pass. =============================================================================== ./tests/servers/test_text_server.h:43: TEST SUITE: [TextServer] TEST CASE: [TextServer] Init, font loading and shaping [TextServer] Unicode identifiers ./tests/servers/test_text_server.h:633: ERROR: CHECK( ts->is_valid_identifier(U"\u0646\u0627\u0645\u0647\u200C\u0627\u06CC") ) is NOT correct! values: CHECK( false ) ./tests/servers/test_text_server.h:634: ERROR: CHECK( ts->is_valid_identifier(U"\u0D26\u0D43\u0D15\u0D4D\u200C\u0D38\u0D3E\u0D15\u0D4D\u0D37\u0D3F") ) is NOT correct! values: CHECK( false ) ./tests/servers/test_text_server.h:635: ERROR: CHECK( ts->is_valid_identifier(U"\u0DC1\u0DCA\u200D\u0DBB\u0DD3") ) is NOT correct! values: CHECK( false ) =============================================================================== ./tests/servers/test_text_server.h:43: TEST SUITE: [TextServer] TEST CASE: [TextServer] Init, font loading and shaping [TextServer] Strip Diacritics ./tests/servers/test_text_server.h:737: ERROR: CHECK( ts->strip_diacritics(U"ٱلسَّلَامُ عَلَيْكُمْ") == U"ٱلسلام عليكم" ) is NOTcorrect! values: CHECK( ٱلسَّلَامُ عَلَيْكُمْ == [{?}, {?}, {?}, {?}, {?}, {?}, {?}, {?}, {?}, {?}, {?}, {?}, {?}] ) =============================================================================== [doctest] test cases: 961 | 959 passed | 2 failed | 1 skipped [doctest] assertions: 2405266 | 2405258 passed | 8 failed | [doctest] Status: FAILURE! ```

To get a proper backtrace you should compile with debug_symbols=yes passed explicitly.

Turning this option on seemed to try to compile everything, including things specified as builtin_*=no, even with separate_debug_symbols=yes. Is there a way to just generate debug symbols for the built code and rely on distro-provided debug symbols for the others?

If you are interested, the MR is here. The Debian GitLab instance is currently being upgraded so it may be offline when you try to visit.

twrightsman commented 2 days ago

For what it's worth, I added static_icu_data=no to the build command but got the same test results.

godotengine / godot