ImportError for libffi.so.7 When Trying to Run PyGObject App

[danyeaw]

Thanks for the great library! :heart:

I am trying to bundle Gaphor with is a PyGObject app in to an AppImage. Everything works fine locally (openSUSE Tumbleweed), but when I build it using Ubuntu 20.04 with the GitHub Actions runner the AppImage fails to load.

dan@localhost:~/Downloads> ./gaphor-2.1.1.dev0+e2c18321.AppImage 
Traceback (most recent call last):
  File "/tmp/.mount_gaphorDjYPzc/opt/python3.9/bin/gaphor", line 6, in <module>
    from gaphor.ui import main
  File "/tmp/.mount_gaphorDjYPzc/opt/python3.9/lib/python3.9/site-packages/gaphor/ui/__init__.py", line 10, in <module>
    import gi
  File "/tmp/.mount_gaphorDjYPzc/opt/python3.9/lib/python3.9/site-packages/gi/__init__.py", line 40, in <module>
    from . import _gi
ImportError: libffi.so.7: cannot open shared object file: No such file or directory

Taking a look inside the AppImage:

dan@localhost:~/Downloads> ./gaphor-2.1.1.dev0+e2c18321.AppImage --appimage-extract | grep libffi
squashfs-root/usr/lib/libffi.so.5

libffi.so.5 is bundled, but not libffi.so.7. I also tried to apt-get install libffi7 on the runner to make sure it is installed.

This is the output of running the build:

python-appimage build app /home/runner/work/gaphor/gaphor/appimage
[2020-12-29 18:59:07,226] EXTRACT  python3.9.1-cp39-cp39-manylinux1_x86_64.AppImage
[2020-12-29 18:59:08,108] BUNDLE   org.gaphor.Gaphor.desktop
[2020-12-29 18:59:08,109] BUNDLE   org.gaphor.Gaphor.png
[2020-12-29 18:59:08,111] BUNDLE   org.gaphor.Gaphor.appdata.xml
[2020-12-29 18:59:12,603] BUNDLE   gaphas==3.0.0b3
[2020-12-29 18:59:13,465] BUNDLE   generic==1.0.0
[2020-12-29 18:59:14,054] BUNDLE   importlib-metadata==3.3.0
[2020-12-29 18:59:14,791] BUNDLE   pycairo==1.20.0
[2020-12-29 18:59:15,318] BUNDLE   pygobject==3.38.0
[2020-12-29 18:59:15,877] BUNDLE   tinycss2==1.1.0
[2020-12-29 18:59:16,539] BUNDLE   typing-extensions==3.7.4.3
[2020-12-29 18:59:17,147] BUNDLE   webencodings==0.5.1
[2020-12-29 18:59:17,786] BUNDLE   zipp==3.4.0
[2020-12-29 18:59:18,425] BUNDLE   gaphor==2.1.1
[2020-12-29 18:59:20,072] BUNDLE   entrypoint.sh
[2020-12-29 18:59:20,084] BUILD    AppDir
[2020-12-29 18:59:20,084] INSTALL  appimagetool from https://github.com/AppImage/AppImageKit/releases/download/continuous

[niess]

Hello,

It is likely that libffi.so.5 shipped in the AppImage comes from the vanilla Python install. If I am correct the ctypes package from the standard library uses it as well.

Which package provides gi? I don't see it explicitly in the list of bundled packages? Is it pygobject? In this case I don't see any binary wheel on PyPI for this package. I only see a tarball. This would imply that the _gi package is built from source by pip BUT this occurs on the Ubuntu 20.04 runner which would explain that _gi is looking explicitly for libffi.so.7, i.e. the runner library it was linked to at compile time. In order that the AppImage runs on many linuses one needs to build this package using a manylinux Docker image, not directly the GitHub runner.

Bundling source packages with C extensions is not yet supported by the python-appimage app builder. Only binary wheels should be used, i.e. packages pre-built using a manylinux Docker image as recommended in PEP 513 and the following.

My apologies since this is not clearly stated in the README.

[danyeaw]

Thanks for the quick response @niess!

Yes pygobject provides gi, and yes it is built from source and linked against the GTK libraries. No problem that this isn't currently supported, definitely understand that this is much more of an advanced use case.

For anyone coming across this issue in the future, I was able to get an AppImage built of a PyGObject app by taking these steps:

1. Use PyInstaller to freeze the app. Make sure that your app runs successfully by running ./dist/app-name/app-name prior to proceeding on.
2. Use a script to search through your app at dist/app-name and remove the excludedfiles. These files are on every Linux computer and shouldn't be packaged in the AppImage.
3. Move your dist/app-name/ directory to dist/AppRun/
4. Copy your app icon to dist/AppRun/org.app.name.png
5. Create dist/AppRun/usr/share/metainfo/ and copy your org.app.name.appdata.xml file there.
6. Create dist/AppRun/usr/share/applications/ and copy your org.app.name.desktop file there.
7. Symlink your org.app.name.desktop file that you copied in the last step to dist/AppRun/org.app.name.desktop
8. Create an AppRun file based on the one in the AppImage SDK. You probably can remove all of the export lines except for setting the PATH with export PATH="${HERE}/${PATH:+:$PATH}". This adds the / directory to the path in the AppImage which is where PyInstaller will put your app-name executable. Place the AppRun script file in dist/AppRun/ and make sure it has chmod +x applied to it so that it is executable.
9. Grab the appdir lint script and run ./appdir-lint.sh dist/AppRun to make sure that all of the previous steps got you to a valid AppImage structure.
10. Download the appimage SDK (appimage-tool)[https://github.com/AppImage/AppImageKit/releases/lateset]
11. Run ./appimagetool-x86_64.AppImage dist/AppRun dist/app-name-x86_64.AppImage to build your AppImage

Probably a blog post is in order :smile:. @amolenaar FYI.

[niess]

@danyeaw Sorry, I'll have to update the README to make this point clear

Thank you for the walk-through. Does the resulting AppImage run on any Linux? E.g. CentOS6? Usually one needs to build the application / package with Docker using a manylinux like image in order to get such binary compatibility? But maybe pyinstaller does this under the hood? Or maybe it fetches highly compatible versions of the various binary deps.

The 1st note of the pyinstaller docs clearly states that the result is (of course) OS dependent. But it does not indicates if its "Linux version/flavour" dependent as well?

[danyeaw]

I didn't think about the compatibility, thanks for pointing that out. I figured if it ran on Ubuntu and openSUSE then it would run on most distros. I spun up a CentOS7 VM and I get the following error when I tried to launch it:

Error loading Python lib 'libpython3.9.so.1.0': dlopen: /lib64/libm.so.6: version GLIBC 2.29 not found.

python-gobject packages weren't released until CentOS8, so I am not sure if I am going to be able to use the manylinux docker image to build this. Maybe if I can get Gaphor running on Ubuntu 14.04 (Trusty) it could be used to improve compatibility using Travis CI (:-1:) or a docker image.

[niess]

@danyeaw Then it looks like pyinstaller is doing the compilation on the host as well. Consequently your AppImage won't run on a Linux that uses an earlier version of GLIBC than the host one (2.29 in your case).

In order to solve this issue you could build the app binaries (C extension modules & shared libraries) with a Docker image of an older Linux distro, e.g. manylinux2010 or manylinux2014 if manylinux1 does not support your app. For example as in the L25-38 of this workflow.

[danyeaw]

@niess manylinux2014 is built on centos7. I just realized that we are using features of GTK 3.24 and centos7 ships with 3.22.