How to run a application inside the build folder, using completely vendored SDL_mixer?

[madebr]

When building a project using vendored SDL2_mixer + vendored dynamically loaded 3rd party libraries, all external modules are built in various subdirectories. Running this program will fail because the system does not know how to find any 3rd party libraries. Runpaths are usually used to accomplish this but because dlopen is in libSDL2.so, the runpath of SDL2_mixer or any program is not taken into account.

Reproducer:

Assume a Linux development machine, having only installed cmake+gcc+ninja+SDL2 (no libmodplug/libxmp/...). Assume you're vendoring SDL_mixer inside your own project. Assume you're using using vendored modules ==> use SDL_mixer's submodules

Paste the following to the bottom of SDL_mixer's CMakeLists.txt.

file(WRITE use_mod_shared_module.c [[
#include "SDL.h"
#include "SDL_mixer.h"

int main() {
  int r = SDL_Init(SDL_INIT_AUDIO);
  if (r != 0) {
    fprintf(stderr, "ERROR: SDL_Init returned %d\n", r);
    return 1;
  }
  r = Mix_Init(MIX_INIT_MOD);
  if (r != MIX_INIT_MOD) {
    fprintf(stderr, "ERROR: MIX_Init returned %d\n", r);
    fprintf(stderr, "err=%s\n", Mix_GetError());
    return 1;
  }
  Mix_Quit();

  SDL_Quit();
}
]])
add_executable(use_mod_test use_mod_shared_module.c)
target_link_libraries(use_mod_test PRIVATE SDL2::SDL2 SDL2_mixer)

Then configure and build the project with -DSDL2MIXER_VENDORED=ON -DSDL2MIXER_MOD_MODPLUG=ON -DSDL2MIXER_MOD_MODPLUG_SHARED=ON.

Running ./use_mod_test gives the following output:

ERROR: MIX_Init returned 0
err=MOD support not available

This is because use_mod_test is unable to find libmodplug.so.1. Adding the library containing libmodplug.so.1 to the run path of either libSDL2_mixer.so and use_mod_test does not fix the issue. Adding it to the runpath of libSDL2.so fixes this. But this is obviously unwanted. Setting LD_LIBRARY_PATH to the location of libmodplug.so.1 also fixes it.

This begs the question of how are external projects supposed to know the location of these vendored libraries? It is possible for the CMake project to provide these location such that external projects can add this to LD_LIBRARY_PATH. The external project would need to add this to every invocation, which is easy to miss.

Another solution would be to move SDL_LoadObject to SDL_mixer (or at least do dlopen inside libSDL_mixer.so. Then, by adding the location of the 3rd party libraries to the runpath of libSDL_mixer.so, the program is able to find its vendored submodules. In this case, external projects require no extra code.

(this question is motivated by https://github.com/libsdl-org/SDL_mixer/issues/448#issuecomment-1251618132)

[madebr]

Let's assume this case is not supported, unless you build with SDL3MIXER_*_SHARED=OFF.

[slouken]

This is a valid point, if you just build SDL_mixer, we should copy the dynamically loaded dependencies into place so they can be found. Visual Studio does this by having an external optional folder that you can drop in, but I'm assuming we can do something smarter with CMake?

[madebr]

The copying of the external dynamic libraries does not fix the issue. Modification of libSDL3.so is required, or LD_LIBRARY_PATH.

Reproducer on my system:

Remove system libxmp library, and build SDL3 + SDL3_mixer with vendored mod support

$ sudo dnf remove libxmp # To make sure a system libxmp is not picked up
$ cmake -S $SDL3_SRC -B /tmp/sdl3_build -GNinja
$ cmake --build /tmp/sdl3_build
$ cmake -S $SDL3MIXER_SRC -B /tmp/sdl3mixer_build  -GNinja -DCMAKE_PREFIX_PATH=/tmp/sdl3_build -DSDL3MIXER_VENDORED=TRUE -DSDL3MIXER_MOD=ON
$ cmake --build /tmp/sdl3mixer_build

With these commands, playing a .mod file will fail:

$ /tmp/sdl3mixer_build/playwave /tmp/TERM.MOD 
INFO: Opened audio at 48000 Hz 32 bit stereo

INFO: Couldn't load /tmp/TERM.MOD: Failed loading libxmp.so.4.6.0: libxmp.so.4.6.0: cannot open shared object file: No such file or directory

The libxmp.so.4.6.0 library can be found in /tmp/sdl3mixer_build/external/libxmp/libxmp.so.4.6.0:

$ find . -iname "libxmp.so.4.6.0"
/tmp/sdl3mixer_build/external/libxmp/libxmp.so.4.6.0

Let's add the path of the libxmp.so library to the rpath of libSDL3_mixer.so and try again:

$ patchelf --add-rpath /tmp/sdl3mixer_build/external/libxmp libSDL3_mixer.so
$ /tmp/sdl3mixer_build/playwave /tmp/TERM.MOD 
INFO: Opened audio at 48000 Hz 32 bit stereo

INFO: Couldn't load /tmp/TERM.MOD: Failed loading libxmp.so.4.6.0: libxmp.so.4.6.0: cannot open shared object file: No such file or directory

Nope, doesn't work: Let's copy libxmp.so.4.6.0 to the same directory as libSDL3_mixer.so, and try again:

$ cp /tmp/sdl3mixer_build/external/libxmp/libxmp.so.4.6.0 /tmp/sdl3mixer_build
$ /tmp/sdl3mixer_build/playwave /tmp/TERM.MOD 
INFO: Opened audio at 48000 Hz 32 bit stereo

INFO: Couldn't load /tmp/TERM.MOD: Failed loading libxmp.so.4.6.0: libxmp.so.4.6.0: cannot open shared object file: No such file or directory

Nope. Let's copy libxmp.so.4.6.0 to /tmp/sdl3_build

$ cp /tmp/sdl3mixer_build/external/libxmp/libxmp.so.4.6.0 /tmp/sdl3_build
$ /tmp/sdl3mixer_build/playwave /tmp/TERM.MOD 
INFO: Opened audio at 48000 Hz 32 bit stereo

INFO: Couldn't load /tmp/TERM.MOD: Failed loading libxmp.so.4.6.0: libxmp.so.4.6.0: cannot open shared object file: No such file or directory

Nope. Let's modify the rpath of libSDL3.so:

$ patchelf --add-rpath /tmp/sdl3mixer_build/external/libxmp /tmp/sdl3_build/libSDL3.so
$ /tmp/sdl3mixer_build/playwave /tmp/TERM.MOD
INFO: Opened audio at 48000 Hz 32 bit stereo

load_module [/home/maarten/programming/SDL_mixer/external/libxmp/src/load.c:248] load
load_module [/home/maarten/programming/SDL_mixer/external/libxmp/src/load.c:253] test Fast Tracker II
load_module [/home/maarten/programming/SDL_mixer/external/libxmp/src/load.c:253] test Amiga Protracker/Compatible
(etc)

This works!

This behavior is caused by the actual dlopen being done in libSDL3.so, and thus the rpath of libSDL3_mixer.so is not taken into account. The search procedure is documented in the man page of dlopen.

Since modifying libSDL3.so is not always possible, and copying the shared libraries does not fix the issue, the only solution I see is to embed SDL_LoadFunction in SDL3_mixer, or running with LD_LIBRARY_PATH=/tmp/sdl3mixer_build/external/libxmp.

[slouken]

Ah, sorry, I was thinking about the macOS and Windows case, where they look for shared libraries bundled with the application by default. For UNIX the dependencies need to be installed before they're found and that's expected behavior.

Games that bundle their own dependencies typically have a launch script that will set the LD_LIBRARY_PATH to the location of the bundled libraries.

By the way, the library that is loaded should be the SONAME, not the fully qualified versioned library. e.g. libxmp.so.4, not libxmp.so.4.6.0 (assuming libxmp.so.4 is a symlink to libxmp.4.6, and that is a symlink to the final version.) This way the system libraries can have their minor or patch versions updated without breaking other applications.

[madebr]

Games that bundle their own dependencies typically have a launch script that will set the LD_LIBRARY_PATH to the location of the bundled libraries.

So I can peacefully close this issue? :)

By the way, the library that is loaded should be the SONAME, not the fully qualified versioned library. e.g. libxmp.so.4, not libxmp.so.4.6.0 (assuming libxmp.so.4 is a symlink to libxmp.4.6, and that is a symlink to the final version.) This way the system libraries can have their minor or patch versions updated without breaking other applications.

I just found out about $<TARGET_SONAME_FILE_NAME:tgt>. I've added it to https://github.com/libsdl-org/SDL_mixer/pull/555.

[slouken]

Games that bundle their own dependencies typically have a launch script that will set the LD_LIBRARY_PATH to the location of the bundled libraries.

So I can peacefully close this issue? :)

I assume so? I was thinking that CMake might have a different workflow since people seem to be able to expect to include different projects and run, but if that's not the case, when we should document that and close this.

[slouken]

I just found out about $<TARGET_SONAME_FILE_NAME:tgt>. I've added it to #555.

This change should be rolled out throughout the SDL satellite libraries (and SDL itself, for that matter, for dynamically loaded libraries)

[madebr]

The fix is in all SDL satellite libraries.

Closing this issue, which is the same as https://github.com/libsdl-org/SDL_image/issues/259 (I think)