AppImages don't work in Docker containers any more

[TheAssassin]

A few weeks ago, I first noticed that some AppImages no longer run in Docker containers. This is a bit annoying especially when working with software like e.g., linuxdeploy, to build AppImages in those containers.

Now, the strange part is that I couldn't reproduce this on Travis yet. But tonight I found out that it must have something to do with the magic bytes in the ELF header. As mentioned previously (e.g., in #699), by embedding the magic bytes as we do currently, we occupy some "might be evaluated by interpreter" ones, which break other software, e.g., qemu. It seems like this is the case in those containers as well.

Temporary fix:

dd if=/dev/zero bs=1 count=3 seek=8 conv=notrunc of=my.AppImage

We should analyze whether this is a bug we caused while we changed the building of the runtime recently (see #818), or whether some component e.g., received an update, in the container. I'm pretty sure I've seen this issue before that PR was merged, though, and actually it'd be better if this was a bug in this container in my opinion.

A start would be to test this in other container images. I ran ubuntu:trusty containers, and we could check whether e.g., centos or opensuse containers yield the issue as well. We could also try this on the latest Ubuntu trusty live ISO.

Note: I tested a variety of AppImages and also the pure runtime, this is not limited to linuxdeploy only.

[luzpaz]

Is this still an issue ?

[probonopd]

I am running AppImages in Docker containers all the time on GitLab CI, it seems to work with --appimage-extract-and-run at least.

[luzpaz]

@probonopd Thanks for that. That is a relief because over at FreeCAD there is an idea of integrating AppImage generation via Docker builds.

[aniongithub]

This is still happening - I cannot use --appimage-extract-and-run for the Unity Hub AppImage and I can't use it from wothin docker at the moment. Help!?

Error: --appimage-extract-and-run is not yet implemented in version continuous-9-gaee2232

[probonopd]

Please ask Unity to use a more recent version of the AppImage tools from https://github.com/AppImage/AppImageKit/releases/tag/continuous. Then the AppImage will support --appimage-extract-and-run. In the meantime, maybe you can use --appimage-extract and the run AppRun manually?

[TheAssassin]

Another workaround: download the raw runtime from the release page @probonopd mentioned, then run like so:

> chmod +x runtime-x86_64
> env TARGET_APPIMAGE=my.AppImage APPIMAGE_EXTRACT_AND_RUN=1 ./runtime-x86_64 ...

[ExpandingMan]

I have been unable to run appimages inside of docker apparently because I can't seem to use FUSE.

I get fuse: device not found, try 'modprobe fuse' first. Needless to say, using modprobe does not work. This may be relevant.

Needless to say, this is pretty disruptive. Requiring different ways to deploy binaries in docker and on hardware is a pretty major obstacle to creating a reproducible setup.

[TheAssassin]

How is it AppImage's fault if your runtime doesn't provide the minimum requirements (which are, by the way, indeed documented, see https://docs.appimage.org/user-guide/troubleshooting.html#i-get-some-errors-related-to-something-called-fuse.

Also, this is not really a new issue. Beginning with Ubuntu xenial, Docker didn't permit FUSE access, as it can be unsafe to let Docker access those kernel features.

Anyway. You can use --appimage-extract-and-run or APPIMAGE_EXTRACT_AND_RUN=1, and it'll work just fine.

Requiring different ways to deploy binaries in docker and on hardware is a pretty major obstacle to creating a reproducible setup.

As said, Docker is not 100% equivalent to a bare metal installation. Your assertation is just incorrect IMO.

[ExpandingMan]

I'm not blaming AppImage, I'm just saying it's an issue.

[TheAssassin]

The runtime should mention the flags more prominently, I think. That'd fix a few issues.

Did you install libfuse, by the way? You will see a different error message if you don't have libfuse installed at all.

[ExpandingMan]

Yup, installed libfuse. As you said, this probably isn't really something appimage can solve. I'll try APPIMAGE_EXTRA_AND_RUN=1. I take it in this case it first caches the entire image in memory?

[TheAssassin]

No, it extracts it to /tmp instead of just FUSE-mounting it.

If you need to access the AppImage more than once, you can also set NO_CLEANUP to any value.

[ExpandingMan]

Cool thanks. Well I feel pretty satisfied with that. It's not ideal, but as we've now both said repeatedly, not really much AppImage can do about it. Some dedicated "Running AppImage in Docker" docs may be in order, but it's entirely possible they exist and I just haven't seen them.

[TheAssassin]

Feel free to open issues or (even better) file PRs to improve the docs.

[mtolly]

I'm still having a problem with this; I'm unable to even run the extract option. This setup definitely used to work for me on a different Linux + Docker install, but now does not (currently on Manjaro with Docker version 19.03.13-ce, build 4484c46d9d):

$ docker run -it ubuntu:14.04
root@87bea274b851:/# apt-get -y install wget
(...)
root@87bea274b851:/# wget https://github.com/probonopd/linuxdeployqt/releases/download/7/linuxdeployqt-7-x86_64.AppImage
(...)
2020-11-22 18:17:20 (19.3 MB/s) - 'linuxdeployqt-7-x86_64.AppImage' saved [14729256/14729256]
root@87bea274b851:/# chmod +x linuxdeployqt-7-x86_64.AppImage 
root@87bea274b851:/# ls    
bin  boot  dev  etc  home  lib  lib64  linuxdeployqt-7-x86_64.AppImage  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@87bea274b851:/# ./linuxdeployqt-7-x86_64.AppImage --appimage-extract
bash: ./linuxdeployqt-7-x86_64.AppImage: No such file or directory

This also happens in ubuntu:18.04, as well as testing other AppImages including one I created previously with linuxdeployqt. The sed trick also does not work because I get a crash when extracting:

root@d6dfdadbf459:/# sed 's|AI\x02|\x00\x00\x00|g' -i linuxdeployqt-7-x86_64.AppImage 
root@d6dfdadbf459:/# ./linuxdeployqt-7-x86_64.AppImage --appimage-extract
squashfs-root/.DirIcon
squashfs-root/AppRun
squashfs-root/linuxdeployqt.desktop
squashfs-root/linuxdeployqt.svg
squashfs-root/usr
squashfs-root/usr/bin
squashfs-root/usr/bin/appimagetool
squashfs-root/usr/bin/desktop-file-validate
squashfs-root/usr/bin/linuxdeployqt
squashfs-root/usr/bin/patchelf
squashfs-root/usr/bin/qt.conf
squashfs-root/usr/bin/zsyncmake
squashfs-root/usr/lib
squashfs-root/usr/lib/appimagekit
squashfs-root/usr/lib/appimagekit/mksquashfs
squashfs-root/usr/lib/libQt5Core.so.5
squashfs-root/usr/lib/libicudata.so.52
sqfs_read_range error: File exists
*** Error in `./linuxdeployqt-7-x86_64.AppImage': corrupted size vs. prev_size: 0x0000000002490de0 ***
Aborted (core dumped)

I can work around this by downloading and extracting the AppImage outside of Docker but it'd be nice to be able to do it inside the Dockerfile. Is there possibly something wrong with my Docker install?

[TheAssassin]

@ExpandingMan see https://docs.appimage.org/user-guide/troubleshooting/fuse.html?highlight=docker#fuse-and-docker.

[mtolly]

Found my fix - the sed command just needed to be modified to only modify the first such occurrence of those bytes, because there was a second one later in the file. I don't know sed so maybe there's a better way but this worked for me.

sed '0,/AI\x02/{s|AI\x02|\x00\x00\x00|}' -i linuxdeployqt-7-x86_64.AppImage

[TheAssassin]

@mtolly Didn't you see that this was suggested in the initial post already?

[mtolly]

@TheAssassin Yes, I read your post. Your original sed command zeroes out all places where that byte combination appears, which can corrupt the program. I tweaked it to only zero out the first one.

[TheAssassin]

You're right. I've updated it (removed the trailing g). Thanks for the hint.

[Adam-]

I've figured out the crash @mtolly has is from this double fclose:

                        while (bytes_already_read < inode.xtra.reg.file_size) {
                            char buf[bytes_at_a_time];
                            if (sqfs_read_range(&fs, &inode, (sqfs_off_t) bytes_already_read, &bytes_at_a_time, buf)) {
                                perror("sqfs_read_range error");
                                fclose(f); <-- here
                                rv = false;
                                break;
                            }
                            // fwrite(buf, 1, bytes_at_a_time, stdout);
                            fwrite(buf, 1, bytes_at_a_time, f);
                            bytes_already_read = bytes_already_read + bytes_at_a_time;
                        }
                        fclose(f);
                        chmod(prefixed_path_to_extract, st.st_mode);
                        if (!rv)
                            break;

But, I do not know why sqfs_read_range is returning an error.

[TheAssassin]

This does look broken indeed. Please send a PR.

[donn]

Interesting to note: CentOS reports this error a bit differently:

/bin/sh: ./appimagetool: /lib64/ld-linux-x86-64.so.2: bad ELF interpreter: No such file or directory
The command '/bin/sh -c ./appimagetool --appimage-extract' returned a non-zero code: 126

Same magic number patching workaround works.

[TheAssassin]

Any of you who have AppImageLauncher installed: please install the latest continuous releases, they contain a brand new workaround that should avoid this issue.