Dependency update: linux release tracking issue

[DarkFenX]

This issue is for tracking major issues which appeared with release of https://github.com/pyfa-org/Pyfa/releases/tag/v2.57.3

Report inability to launch pyfa, crashes, weird looking UI here.

Changes: dependencies updated, and AppImage build process was changed. Since it now includes minimal amount of dependencies (which was just enough to make it work on installations I had access to), it's very likely that AppImage won't launch on some installations due to missing libraries. Those issues are especially important to know.

[wereii]

ArchLinux 6.6.3, Python3.11.6

> ./pyfa-v2.57.0dev1-linux.AppImage
Traceback (most recent call last):
  File "/tmp/.mount_pyfa-vDiRD1D/opt/pyfa/pyfa.py", line 27, in <module>
    import config
  File "/tmp/.mount_pyfa-vDiRD1D/opt/pyfa/config.py", line 4, in <module>
    import wx
  File "/tmp/.mount_pyfa-vDiRD1D/usr/local/lib/python3.11/dist-packages/wx/__init__.py", line 17, in <module>
    from wx.core import *
  File "/tmp/.mount_pyfa-vDiRD1D/usr/local/lib/python3.11/dist-packages/wx/core.py", line 12, in <module>
    from ._core import *
ImportError: /usr/lib/libtiff.so.6: undefined symbol: jpeg12_write_raw_data, version LIBJPEG_8.0

[DarkFenX]

@wereii judging by path, it seems to prefer system-provided libtiff over bundled one. I don't know what to do about it yet.

Oh, another thing is that due to build time limits linux build doesn't compile wx by itself, but gets binary package from wx site, which can cause its own set of issues (previously it was commented out by blitzmann as causing compatibility issues).

edit: https://github.com/audacity/audacity/issues/5168 might give some info on root cause and maybe on how to fix it (will look into it later)

[DarkFenX]

@wereii output of LD_DEBUG=libs ./pyfa-v2.57.0dev1-linux.AppImage please

[wereii]

Here you go https://termbin.com/l68s

Yeah building from pre-build wxPython is something I think I added back then when we were dealing with the Arch incompatibility and it caused that the appimage simply did not work on some distros.

I could look into building (and hosting) appimage-friendly wxPython in a "side" repo...

[DarkFenX]

Ok im blind and confused libtiff versions (we bundle libtiff5, error is about libtiff6). Looks like something requests to load libtiff6, which is provided only by your system, and it tries to use libjpeg function which is not defined in bundled libjpeg version. So, need to figure out dependency chain, and bundle top-level library which needs that with AppImage

[DarkFenX]

I have no idea how to debug it properly (w/o having arch at hand), so I just checked arch repo and found one of libs which were loaded earlier which depend on libtiff. Looks like it's this package: https://archlinux.org/packages/extra/x86_64/gdk-pixbuf2/

To prevent those issues, I think I will have to bundle gtk3, which might increase distributive size by a lot.

[wereii]

I do have gdk-pixbuf2 installed, /usr/lib/libtiff.so.6 is provided by libtiff package - so probably wxPython or one of it's dependencies in the appimage have been built with libtiff.so.6 but the same lib on Arch does not conatain same symbols, that's a bummer.

You could try making a symlink in the appimage from libtiff.so.6 to libtiff.so.5 (hacky I know).

[DarkFenX]

I think it's error-prone. I will just bundle needed libs in appimage.

[DarkFenX]

@wereii try updated release please https://github.com/pyfa-org/Pyfa/releases/tag/v2.57.0dev2

edit: also, gtk is fat, and so are wx-html dependencies (~100 uncompressed bm just to show update notifications)

[wereii]

https://termbin.com/jtia

Pyfa now starts but there are some errors right at the start.
Just going thru the File, Fit, Character, ... tabs there is a lot of the GLib-GObject-CRITICAL errors generated, guessing something with the images in the menus.

And Opening Global > Preferences or Editors > Character Editor crashes pyfa with the:

Bail out! Gtk:ERROR:../../../../gtk/gtkiconhelper.c:494:ensure_surface_for_gicon: assertion failed (error == NULL): Failed to load /org/gtk/libgtk/icons/16x16/status/image-missing.png: Unrecognized image file format (gdk-pixbuf-error-quark, 3)
[1]    20009 IOT instruction (core dumped)  ./pyfa-v2.57.0dev2-linux.AppImage 2> log.txt

What works (and I tried): graphs, fitting mid slots/removing from slots, implant editor.

[DarkFenX]

Can you upload /org/gtk/libgtk/icons/16x16/status/image-missing.png somewhere? (looks like it's trying to load system one)

Also, strace -o strace.log ./pyfa-v2.57.0dev2-linux.AppImage for both cases: preferences and char editor (no idea what I can extract from that, but maybe will find some clues).

[wereii]

This path does not exist as is (there is no root /org directory by default on probably any linux).
I did some searches for the file name but no hits both in pyfa files and outside, I will keep looking but I have to go now, be back in about an hour.

[wereii]

So according to this https://gitlab.gnome.org/GNOME/gimp/-/issues/7337 the path in question should be a "compiled" resource.

I had a look around for same errors and arrived to this thread https://github.com/bottlesdevs/Bottles/issues/230#issuecomment-1737248279 - trying the fix in the linked comment (without the root path, that seems wrong):

XDG_DATA_DIRS="$HOME/.local/share:/usr/local/share/:/usr/share/" ./pyfa-v2.57.0dev2-linux.AppImage

I get pyfa running without any errors, the tabs that crashed the app before work alright now, the icons in menus are back and they are even the right theme (so instead of white windows I get same theme colors as in other GTK apps).

Though now I am unsure if this wasn't just a misconfiguration on my part.

[DarkFenX]

What's XDG_DATA_DIRS's value without override?

I think it's hard to say if it's your issue or package's. I think it can be resolved on both sides, but idk what the implications would be.

Also, apparently current image contains 25 uncompressed mb of icons, and I am unsure if those can be excluded.

[DarkFenX]

Theoretically I can add paths to bundled icons to the end of the variable, then there is sensible fallback in case your system variable is misconfigured.

[DarkFenX]

https://docs.gtk.org/gtk3/running.html those are the variables which count. Will play with those and ping you again once I have new build.

AppImage on my system ignores system theme setting btw, running it as GTK_THEME=Adwaita:dark ./pyfa-v2.57.0dev2+5-g89f1e0a12-linux.AppImage works fine. So there is definitely some env variable fine-tuning left to do.

[wereii]

What's XDG_DATA_DIRS's value without override?

Empty. According to a quick search, apps can fallback to a given default but not every app does? I have not noticed any problems with other gtk apps without this value.

The whole XDG is notoriously confusing, at least from my experience.

There seems to be some talk about defaulting or not for these variables: https://github.com/AppImage/AppImageKit/issues/481

Pyfa could set the defaults in the entry script of the appimage, https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html:

If $XDG_DATA_DIRS is either not set or empty, a value equal to /usr/local/share/:/usr/share/ should be used.

export XDG_DATA_DIRS="${XDG_DATA_DIRS:-/usr/local/share/:/usr/share/}" 

this should make sure the variables are always set to either user or default value. (the "set default" syntax is explained here https://stackoverflow.com/a/2013589/588179)

[DarkFenX]

will it work fine with base empty value though? (when value starts from :, not first path, I never ran into this situation)

edit: nvm you don't append, I see

[DarkFenX]

I can't seem to find a way to set variables this way according to those docs: https://appimage-builder.readthedocs.io/en/latest/reference/recipe.html

So honestly it's probably better to just assume configuration is properly set already instead of trying to fix it on pyfa side.

[wereii]

Ah, I was thinking that the entry point of the AppImage could be a shell script exporting the vars then execing python3.11 ... .../.../pyfa.py but I see there is now AppImageBuilder.yml - not familiar with that one.

I can handle "fixing" XDG_ vars for Arch in the AUR package (by doing what I have described above with the wrapping shell script) so I guess it's not really required (except as you said, it seems to ignore gtk theme without it?)

[DarkFenX]

If you think your OS has nothing to do with setting those variables, I still might try to play with setting them in AppImageBuilder.yml. But if I do that, they will be pointing to bundled resources, not to system ones (since, well, system has to handle that). All I need to do is to find a way to append bundled paths to variable, no matter if it's defined or not, which will also preferably work in env variables section w/o extra hacks.

[wereii]

My current theory is that apps using these vars should be ready to use the freedesktop defaults if the vars are unset/empty (which in this case actually applies to the deps of pyfa, wxPython and lower) but I do not have a concrete information to show so for now let's leave it out, but if someone else with different distro gets the same error then we have a confirmation.

[DarkFenX]

Are there any docs on what defaults should be? I guess I can set those for system as priority, then similar but bundled paths as fallback

[wereii]

https://wiki.archlinux.org/title/XDG_Base_Directory this seems to source the information from here (which is the offical doc for XDG) https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html (the defaults for each variable are mentioned in the text).

[DarkFenX]

Ok so let's say I want to do following things:

1) If XDG_DATA_DIRS is empty, set it to /usr/local/share:/usr/share 2) No matter if it was empty or not, append :${APPDIR}/usr/share to it

The issue is, I have only 1 step per variable to do that (AppDir/runtime/env is a YAML mapping), is there a way to do that in bash?

[wereii]

yes it's possible in bash, or just about any shell, afaik the set-default expansion is generic for any shell implementation

# set default if not exists
XDG_DATA_DIRS="${XDG_DATA_DIRS:-/usr/local/share/:/usr/share/}" 
# append AppImage location as fallback
XDG_DATA_DIRS="$XDG_DATA_DIRS:$APPDIR/usr/share"
# set globally
export XDG_DATA_DIRS

[DarkFenX]

Well, it has to be a one-liner, I can't do multiple steps in here: https://github.com/pyfa-org/Pyfa/blob/346184d06a20a8b48a85192c159c94195e8f7168/dist_assets/linux/AppImageBuilder.yml#L63

anyway, for now I think setting default should be good (as long as it's interpreted properly by the builder), please check it: https://ci.appveyor.com/project/pyfa-org/pyfa/builds/48687116/job/rec55i5t63sspxr9/artifacts

[wereii]

one-liner

right, this miight work: ${XDG_DATA_DIRS:-/usr/local/share/:/usr/share/}:${APPDIR}/usr/share

e: not at my linux machine rn, will check in an hour or some

[DarkFenX]

Build with your suggested variant: https://ci.appveyor.com/project/pyfa-org/pyfa/builds/48687316/job/4031v58mpmv8j23w/artifacts

Please test it first. If it doesn't work test previous one.

[wereii]

Ah, tried downloading the first one but my browser crashed before it got it whole and now I am getting error {"message":"Artifacts download limit (1024 MB/day) exceeded."}

e: it seems to be IP based, so the second one returns the same

[DarkFenX]

It's probably project-based, but yes. Idk i can get a build out (via making a draft release) but no idea where to upload it

[wereii]

I've sent you email (to the address on your gh profile) I could get some storage for you to upload to in few minutes if the wormhole approach does not work.

e: got the file e2: still crashes

[DarkFenX]

But now it also crashes on my laptop:

Gtk:ERROR:../../../../gtk/gtkiconhelper.c:494:ensure_surface_for_gicon: assertion failed (error == NULL): Failed to load /org/gtk/libgtk/icons/16x16/status/image-missing.png: Unrecognized image file format (gdk-pixbuf-error-quark, 3)
Bail out! Gtk:ERROR:../../../../gtk/gtkiconhelper.c:494:ensure_surface_for_gicon: assertion failed (error == NULL): Failed to load /org/gtk/libgtk/icons/16x16/status/image-missing.png: Unrecognized image file format (gdk-pixbuf-error-quark, 3)
Aborted

[DarkFenX]

Judging by this file: https://github.com/AppImageCrafters/appimage-builder/blob/main/appimagebuilder/modules/setup/apprun_2/apprun2.py

appimage-builder doesn't use bash to interpret env variables. For the variable in question it takes its value as string in parse_env_input() (with variable substitution before that, i think), then in _configure_runtime_environment() it just sets them. There is no extra fancy stuff.

In the latter function btw there is a block:

        global_env = Environment(
            {
                "APPIMAGE_UUID": bundle_id,
                "XDG_DATA_DIRS": [
                    "$APPDIR/usr/local/share",
                    "$APPDIR/usr/share",
                    "$XDG_DATA_DIRS",
                ],
                "XDG_CONFIG_DIRS": ["$APPDIR/etc/xdg", "$XDG_CONFIG_DIRS"],
                "APPDIR_LIBRARY_PATH": self._get_appdir_library_paths(),
                "PATH": [*self._get_bin_paths(), "$PATH"],
            }
        )

Which means that system default paths are not used by default (for bundled software it sets paths, so it never receives empty variable; but even if it was empty, bundled software doesn't know much about system paths i imagine), which also means that paths need to be set explicitly. This seems to be a bug in appimage-builder really, since it needs to add system default paths if the variable is empty.

For now, I will remove variable override, and add issue in the appimage-builder repo.

edit: created issue https://github.com/AppImageCrafters/appimage-builder/issues/336

[DarkFenX]

Theoretically we can take definition of XDG_DATA_DIRS as specified in _configure_runtime_environment(), but also add system paths to it (to make sure they are always there, even if duplicated).

If the var is set, I end up with this: /tmp/.mount_pyfa-vSEGTpW/usr/local/share:/tmp/.mount_pyfa-vSEGTpW/usr/share:/usr/local/share:/usr/share:/usr/share/gnome:/home/dfx/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share/:/usr/share/

If var isn't set or empty, it looks like this: /tmp/.mount_pyfa-vqKkw0R/usr/local/share:/tmp/.mount_pyfa-vqKkw0R/usr/share:/usr/local/share:/usr/share:

So, should work on your system

[wereii]

Yes, confirming it now works without crashes or the errors generated into terminal. :tada:

[willl03]

Hi, when I try to run pyfa in WSL 2 Ubuntu 22.04 it fails with the following error - np before the dependency update.

./pyfa.AppImage Traceback (most recent call last): File "/tmp/.mount_pyfa.AHK2JFE/opt/pyfa/pyfa.py", line 27, in import config File "/tmp/.mount_pyfa.AHK2JFE/opt/pyfa/config.py", line 4, in import wx File "/tmp/.mount_pyfa.AHK2JFE/usr/local/lib/python3.11/dist-packages/wx/init.py", line 17, in from wx.core import File "/tmp/.mount_pyfa.AHK2JFE/usr/local/lib/python3.11/dist-packages/wx/core.py", line 12, in from ._core import ImportError: /tmp/.mount_pyfa.AHK2JFE/usr/lib/x86_64-linux-gnu/libgdk-3.so.0: undefined symbol: wl_proxy_marshal_flags

[DarkFenX]

We don't package all the dependencies in appimage, some are still provided by the system. In this case it seems that libgdk doesn't have its dependency bundled in the appimage, and it can't find needed functionality in your system libraries. But this is weird because ubuntu 22.04 is used as build system, so idk why this mismatch happens.

Dependency of the package: https://packages.ubuntu.com/jammy/libgtk-3-0 A bit more useful context: https://github.com/libsdl-org/SDL/issues/5376#issuecomment-1075589041

@willl03 could you update your system (one of libwayland-* libs) and see if issue persists?

[engineer1109]

Hi, when I try to run pyfa in WSL 2 Ubuntu 22.04 it fails with the following error - np before the dependency update.

./pyfa.AppImage Traceback (most recent call last): File "/tmp/.mount_pyfa.AHK2JFE/opt/pyfa/pyfa.py", line 27, in import config File "/tmp/.mount_pyfa.AHK2JFE/opt/pyfa/config.py", line 4, in import wx File "/tmp/.mount_pyfa.AHK2JFE/usr/local/lib/python3.11/dist-packages/wx/init.py", line 17, in from wx.core import File "/tmp/.mount_pyfa.AHK2JFE/usr/local/lib/python3.11/dist-packages/wx/core.py", line 12, in from ._core import ImportError: /tmp/.mount_pyfa.AHK2JFE/usr/lib/x86_64-linux-gnu/libgdk-3.so.0: undefined symbol: wl_proxy_marshal_flags

Me too

[DarkFenX]

Let's use #2600 for marshal_flags issue. I am closing this generic one.