LagoLunatic / wwrando

Wind Waker Randomizer
https://lagolunatic.github.io/wwrando/
MIT License
383 stars 85 forks source link

Linux builds from github actions mix vendored and system libraries #270

Closed Natolumin closed 2 years ago

Natolumin commented 2 years ago

I've been running into issues with the linux builds from the development version as built from the github actions, with running in X11 causing segfaults (I can reproduce this on both fedora 35 and gentoo) Running from source and running from an executable built locally with pyinstaller works as expected. I expect that it also runs fine on ubuntu 20.04 or systems with mostly identical libraries since that's what the build environment in github actions runs on

I think I have tracked this to the build environment in the github actions VMs missing some libraries (as seen in the pyInstaller logs here), which causes execution to mix and match some bundled libraries with system libraries on the executing system : the libaries that aren't found and vendored at build time get loaded from the system where the binary runs if available, but they might not have exactly compatible versions. In my particular case this appears to be from libxkbcommon libs and the mix causes a segfault, but I imagine this could cause other issues with other libraries in the same situation

Some stacktraces and ldd output for more detail ldd on one of the bundled Qt libraries shows it trying to load libxkbcommon from its own set of bundled libraries in `/tmp/_MEISFMUPJ ` (from ubuntu focal that would be from libxkbcommon 0.10.0) and libxkbcommon-x11 from my system (which has libxkbcommon 1.4.0): ``` /tmp/_MEISFMUPJ $ QT_QPA_PLATFORM=xcb lddtree libQt6XcbQpa.so.6 libQt6XcbQpa.so.6 (interpreter => None) libxcb-icccm.so.4 => /usr/lib64/libxcb-icccm.so.4 libxcb-image.so.0 => /usr/lib64/libxcb-image.so.0 libxcb-util.so.1 => /usr/lib64/libxcb-util.so.1 libxcb-keysyms.so.1 => /usr/lib64/libxcb-keysyms.so.1 libxcb-randr.so.0 => /usr/lib64/libxcb-randr.so.0 libxcb-render.so.0 => /tmp/_MEISFMUPJ/libxcb-render.so.0 libxcb-render-util.so.0 => /usr/lib64/libxcb-render-util.so.0 libxcb-shape.so.0 => /tmp/_MEISFMUPJ/libxcb-shape.so.0 libxcb-shm.so.0 => /tmp/_MEISFMUPJ/libxcb-shm.so.0 libxcb-sync.so.1 => /tmp/_MEISFMUPJ/libxcb-sync.so.1 libxcb.so.1 => /usr/lib64/libxcb.so.1 libXau.so.6 => /tmp/_MEISFMUPJ/libXau.so.6 libXdmcp.so.6 => /tmp/_MEISFMUPJ/libXdmcp.so.6 libbsd.so.0 => /tmp/_MEISFMUPJ/libbsd.so.0 libxcb-xfixes.so.0 => /tmp/_MEISFMUPJ/libxcb-xfixes.so.0 libxcb-xkb.so.1 => /usr/lib64/libxcb-xkb.so.1 libxkbcommon.so.0 => /tmp/_MEISFMUPJ/libxkbcommon.so.0 libglib-2.0.so.0 => /tmp/_MEISFMUPJ/libglib-2.0.so.0 libpcre.so.3 => /tmp/_MEISFMUPJ/libpcre.so.3 libX11-xcb.so.1 => /tmp/_MEISFMUPJ/libX11-xcb.so.1 libX11.so.6 => /tmp/_MEISFMUPJ/libX11.so.6 libdl.so.2 => /lib64/libdl.so.2 libxkbcommon-x11.so.0 => /usr/lib64/libxkbcommon-x11.so.0 libQt6OpenGL.so.6 => /tmp/_MEISFMUPJ/libQt6OpenGL.so.6 libQt6Gui.so.6 => /tmp/_MEISFMUPJ/libQt6Gui.so.6 libEGL.so.1 => /usr/lib64/libEGL.so.1 libGLdispatch.so.0 => /usr/lib64/libGLdispatch.so.0 libfontconfig.so.1 => /tmp/_MEISFMUPJ/libfontconfig.so.1 libexpat.so.1 => /tmp/_MEISFMUPJ/libexpat.so.1 libuuid.so.1 => /tmp/_MEISFMUPJ/libuuid.so.1 ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 libQt6DBus.so.6 => /tmp/_MEISFMUPJ/libQt6DBus.so.6 libdbus-1.so.3 => /tmp/_MEISFMUPJ/libdbus-1.so.3 libsystemd.so.0 => /tmp/_MEISFMUPJ/libsystemd.so.0 librt.so.1 => /lib64/librt.so.1 liblzma.so.5 => /tmp/_MEISFMUPJ/liblzma.so.5 liblz4.so.1 => /tmp/_MEISFMUPJ/liblz4.so.1 libgcrypt.so.20 => /tmp/_MEISFMUPJ/libgcrypt.so.20 libgpg-error.so.0 => /tmp/_MEISFMUPJ/libgpg-error.so.0 libz.so.1 => /tmp/_MEISFMUPJ/libz.so.1 libfreetype.so.6 => /tmp/_MEISFMUPJ/libfreetype.so.6 libpng16.so.16 => /tmp/_MEISFMUPJ/libpng16.so.16 libGL.so.1 => /usr/lib64/libGL.so.1 libGLX.so.0 => /usr/lib64/libGLX.so.0 libQt6Core.so.6 => /tmp/_MEISFMUPJ/libQt6Core.so.6 libicui18n.so.56 => /tmp/_MEISFMUPJ/libicui18n.so.56 libicuuc.so.56 => /tmp/_MEISFMUPJ/libicuuc.so.56 libicudata.so.56 => /tmp/_MEISFMUPJ/libicudata.so.56 libpthread.so.0 => /lib64/libpthread.so.0 libgthread-2.0.so.0 => /tmp/_MEISFMUPJ/libgthread-2.0.so.0 libstdc++.so.6 => /tmp/_MEISFMUPJ/libstdc++.so.6 libm.so.6 => /lib64/libm.so.6 libgcc_s.so.1 => /tmp/_MEISFMUPJ/libgcc_s.so.1 libc.so.6 => /lib64/libc.so.6 ``` And this causes this segfault on startup : ``` (gdb) bt #0 0x00007ffff193a324 in atom_intern (table=0xbfac70, string=, len=5, add=true) at ../libxkbcommon-1.4.0/src/atom.c:175 #1 0x00007ffff1939fe5 in xkb_atom_intern (ctx=0x0, string=0xb7bee82e , len=5) at ../libxkbcommon-1.4.0/src/context-priv.c:62 #2 update_builtin_keymap_fields (keymap=0xbfc3c0) at ../libxkbcommon-1.4.0/src/keymap-priv.c:47 #3 xkb_keymap_new (ctx=, format=XKB_KEYMAP_FORMAT_TEXT_V1, flags=XKB_KEYMAP_COMPILE_NO_FLAGS) at ../libxkbcommon-1.4.0/src/keymap-priv.c:72 #4 0x00007ffff1937686 in xkb_x11_keymap_new_from_device (ctx=0xbfb920, conn=0xbd3110, device_id=3, flags=(unknown: 0xf1935da0)) at ../libxkbcommon-1.4.0/src/x11/keymap.c:1156 #5 0x00007fffeef99570 in ?? () from /tmp/_MEISFMUPJ/libQt6XcbQpa.so.6 #6 0x00007fffeef78e78 in QXcbConnection::QXcbConnection(QXcbNativeInterface*, bool, unsigned int, char const*) () from /tmp/_MEISFMUPJ/libQt6XcbQpa.so.6 #7 0x00007fffeef96541 in QXcbIntegration::QXcbIntegration(QList const&, int&, char**) () from /tmp/_MEISFMUPJ/libQt6XcbQpa.so.6 #8 0x00007ffff3de942f in ?? () from /tmp/_MEISFMUPJ/PySide6/Qt/plugins/platforms/libqxcb.so #9 0x00007ffff5bec0c5 in ?? () from /tmp/_MEISFMUPJ/libQt6Gui.so.6 #10 0x00007ffff5beeb8e in QGuiApplicationPrivate::createPlatformIntegration() () from /tmp/_MEISFMUPJ/libQt6Gui.so.6 #11 0x00007ffff5bef630 in QGuiApplicationPrivate::createEventDispatcher() () from /tmp/_MEISFMUPJ/libQt6Gui.so.6 #12 0x00007ffff554cfa6 in QCoreApplicationPrivate::init() () from /tmp/_MEISFMUPJ/libQt6Core.so.6 #13 0x00007ffff5bf12f0 in QGuiApplicationPrivate::init() () from /tmp/_MEISFMUPJ/libQt6Gui.so.6 #14 0x00007ffff061c019 in QApplicationPrivate::init() () from /tmp/_MEISFMUPJ/libQt6Widgets.so.6 #15 0x00007ffff0da4eb5 in QApplicationConstructor(_object*, _object*, QApplicationWrapper**) () from /tmp/_MEISFMUPJ/PySide6/QtWidgets.abi3.so #16 0x00007ffff0daa288 in Sbk_QApplication_Init () from /tmp/_MEISFMUPJ/PySide6/QtWidgets.abi3.so #17 0x00007ffff7b3b96f in type_call (type=, type@entry=0xa748d0, args=args@entry=0x7ffff6e6bac0, kwds=kwds@entry=0x0) at Objects/typeobject.c:1026 #18 0x00007ffff7b04f20 in _PyObject_MakeTpCall (tstate=0x4365c0, callable=0xa748d0, args=0x7ffff70b8f40, nargs=1, keywords=0x0) at Objects/call.c:191 #19 0x00007ffff7b6a616 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7ffff70b8f40, callable=0xa748d0, tstate=) at ./Include/cpython/abstract.h:116 #20 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7ffff70b8f40, callable=0xa748d0, tstate=) at ./Include/cpython/abstract.h:103 #21 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7ffff70b8f40, callable=0xa748d0) at ./Include/cpython/abstract.h:127 #22 call_function (kwnames=0x0, oparg=, pp_stack=, tstate=0x4365c0) at Python/ceval.c:5077 #23 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at Python/ceval.c:3520 #24 0x00007ffff7b645a3 in _PyEval_EvalFrame (throwflag=0, f=0x7ffff70b8dd0, tstate=0x4365c0) at ./Include/internal/pycore_ceval.h:40 #25 _PyEval_EvalCode (qualname=0x0, name=, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2, kwcount=, kwargs=0x0, kwnames=0x0, argcount=, args=, locals=, globals=, _co=, tstate=0x4365c0) at Python/ceval.c:4329 #26 _PyEval_EvalCodeWithName (_co=, globals=, locals=, args=, argcount=, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4361 #27 0x00007ffff7b64297 in PyEval_EvalCodeEx (_co=, globals=, locals=, args=, argcount=, kws=, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:4377 #28 0x00007ffff7beadcf in PyEval_EvalCode (co=, globals=, locals=) at Python/ceval.c:828 #29 0x0000000000403c2f in ?? () #30 0x0000000000403fce in ?? () #31 0x00007ffff7dad34a in __libc_start_call_main (main=main@entry=0x402510, argc=argc@entry=1, argv=argv@entry=0x7fffffffd7c8) at ../sysdeps/nptl/libc_start_call_main.h:58 #32 0x00007ffff7dad3fc in __libc_start_main_impl (main=0x402510, argc=1, argv=0x7fffffffd7c8, init=, fini=, rtld_fini=, stack_end=0x7fffffffd7b8) at ../csu/libc-start.c:392 #33 0x000000000040253e in ?? () ```

I have managed to reproduce this by building with pyinstaller on a basic ubuntu 20.04 vm and running the binary on fedora 35.

A possible fix could be installing qt5-default on the ubuntu vm in the github actions environment before running the pyinstaller build. That package pulls all the same dependencies as qt6 would (PySides6 uses Qt6 but ubuntu 20.04 doesn't package that. In jammy which should be the next LTS there seems to be a qt6-base-dev package). That seems to solve the issue and all transitively required libraries are properly pulled into the binary, and that one runs fine on all systems I've tried it on

LagoLunatic commented 2 years ago

Can you check if this build works now? https://nightly.link/LagoLunatic/wwrando/actions/artifacts/205092573.zip

Natolumin commented 2 years ago

It does! thanks for the quick fix