Exporting GDExtensions doesn't work

[BastiaanOlij]

Godot version

4.0.dev c2a616f3ecc5fa0ee7d85507b971e7578000a562

System information

Windows (but likely other platforms too)

Issue description

When exporting projects that contain GDExtension plugins the dynamic libraries are not exported properly. They can't be loaded from inside of the .pck file and thus need to exist besides the executable.

Manually copying them in place doesn't work either because Godot still attempts to load them from their original location.

For GDNative we had things in place to deal with this on export, need to look into reproducing the same for GDExtensions

Steps to reproduce

Build the test project in godot-cpp Add an export template for Windows Export the test project, see if it runs.

Minimal reproduction project

https://github.com/godotengine/godot-cpp/tree/master/test

[MadMcCrow]

I can confirm it affects other platforms too. I can reproduce on LinuxX11 with godot 4.0 6d930bd270.

[underdoeg]

somewhat off topic, but with android it seems to work. Although I had to manually add the aar file to the android custom build template

[akien-mga]

Should be re-evaluated in 4.0 beta 8 after #67906. This might have been fixed some time ago, and/or fixed by #67906, and/or regressed. We should check what's the status today.

[cridenour]

@akien-mga When manually listing [libraries] exporting continues to work (it has for a while now). However when trying the new autodetect_library_prefix instead, the file is exported next to the executable but can never be found when the game tries to load the extension.

[akien-mga]

@cridenour Could you open a new issue for this?

This issue here was originally for a different problem which has been solved a while ago AFAICT.

[ulvido]

exactly same problem. in editor debug run seems perfect but exported for linux doesn't even recognize my classes. no errors just silently nonexistent. export process copies the lib to the export folder but not imports into the pck.

godot: 4.1.1-stable steam linux: pop_os

[opyate]

Same problem here: https://github.com/opyate/godot-llm-experiment

./Godot\ LLM\ Experiment.x86_64 
ERROR: Can't open dynamic library: bin/libgdllm.linux.template_debug.x86_64.so. Error: libllama.so: cannot open shared object file: No such file or directory.
   at: open_dynamic_library (drivers/unix/os_unix.cpp:650)
ERROR: GDExtension dynamic library not found: bin/libgdllm.linux.template_debug.x86_64.so
   at: open_library (core/extension/gdextension.cpp:455)
ERROR: Failed loading resource: res://bin/gdllm.gdextension. Make sure resources have been imported by opening the project in the editor at least once.
   at: _load (core/io/resource_loader.cpp:273)
ERROR: Error loading extension: res://bin/gdllm.gdextension
   at: load_extensions (core/extension/gdextension_manager.cpp:143)
Godot Engine v4.1.1.stable.official.bd6af8e0e - https://godotengine.org
/lib/x86_64-linux-gnu/libxkbcommon.so.0: undefined symbol: xkb_utf32_to_keysym
/lib/x86_64-linux-gnu/libxkbcommon.so.0: undefined symbol: xkb_keymap_key_get_mods_for_level
OpenGL API 3.3.0 NVIDIA 530.30.02 - Compatibility - Using Device: NVIDIA - NVIDIA RTX 6000 Ada Generation

SCRIPT ERROR: Parse Error: Identifier "GDLLM" not declared in the current scope.
          at: GDScript::reload (res://scenes/dialogue.gd:12)
ERROR: Cannot get class 'GDExample'.
   at: instantiate (core/object/class_db.cpp:339)
WARNING: Node GDExample of type GDExample cannot be created. A placeholder will be created instead.
     at: instantiate (scene/resources/packed_scene.cpp:205)
SCRIPT ERROR: Invalid call. Nonexistent function 'get_next_person_line' in base 'Node2D'.
          at: next_person_say_something (res://scenes/start.gd:18)

I don't see my .so and other binaries in the list:

I expect all the files in bin/ to be visible and exportable:

% ls -1 the-game/bin 
gdllm.gdextension
libgdllm.linux.template_debug.x86_64.so
libllama.so
mistral-7b-instruct-v0.1.Q5_K_M.gguf

[ulvido]

exactly same problem. in editor debug run seems perfect but exported for linux doesn't even recognize my classes. no errors just silently nonexistent. export process copies the lib to the export folder but not imports into the pck.

godot: 4.1.1-stable steam linux: pop_os

my problem was my mistake. I should've registered the extension as MODULE_INITIALIZATION_LEVEL_SCENE bu I mistakenly wrote MODULE_INITIALIZATION_LEVEL_EDITOR. so it works in the editor fine but not works for the build. makes sense.

correct initialization below:

[opyate]

Turns out my problem was my mistake too (or perhaps just a mistaken expectation of what export does). The contents of my bin/ folder doesn't seem to be packed, so I just need to publish it alongside my game binary.

In fact, it would be awesome if someone could explain why the log says it's "storing" my LLM binary, but still the exported game only runs if the bin is alongside it.

https://github.com/opyate/godot-llm-experiment/blob/main/docs/export-log.txt#L377

savepack: step 82: Storing File: res://bin/libgdllm.linux.template_debug.x86_64.so
savepack: step 88: Storing File: res://bin/libllama.so
savepack: step 95: Storing File: res://bin/mistral-7b-instruct-v0.1.Q5_K_M.gguf   # <------ this file

It's a 4.8GB file, if that's got anything to do with it:

% ls -lah the-game/bin 
total 4.8G
drwxrwxr-x 2 opyate opyate 4.0K Oct  9 00:02 .
drwxrwxr-x 6 opyate opyate 4.0K Oct 11 12:08 ..
-rw-rw-r-- 1 opyate opyate 1.3K Oct  6 21:30 gdllm.gdextension
-rwxrwxr-x 1 opyate opyate 6.2M Oct  9 00:02 libgdllm.linux.template_debug.x86_64.so
-rwxrwxr-x 1 opyate opyate 1.1M Oct  7 21:23 libllama.so
-rw-rw-r-- 1 opyate opyate 4.8G Oct  7 21:09 mistral-7b-instruct-v0.1.Q5_K_M.gguf