AppImage(s) on NixOS (doesn't work)

olejorgenb commented 7 years ago

I was curious of whether appimages would run out-of-the box on NixOS since it is set up in a rather special way.

Downloaded the poster image from appimage.org:

$ wget https://subsurface-divelog.org/downloads/Subsurface-4.6.2-x86_64.AppImage
$ chmod +x Subsurface-4.6.2-x86_64.AppImage
$ ./Subsurface-4.6.2-x86_64.AppImage
zsh: no such file or directory: ./Subsurface-4.6.2-x86_64.AppImage

Some info:

$ readelf -a Subsurface-4.6.2-x86_64.AppImage | wgetpaste
https://paste.pound-python.org/show/c2Id8UAY6irakarzPFI6/

$ uname -a
Linux lapole 4.9.45 #1-NixOS SMP Fri Aug 25 00:12:55 UTC 2017 x86_64 GNU/Linux

$ nixos-version
17.03.1775.56da88a298 (Gorilla)

Weirdly(?) ldd says "not a dynamic executable"

LD_LIBRARY_PATH

The weird "no such file ..." message is expected on nixos since /lib64/ld-linux-x86-64.so.2 doesn't exist. We can get hold of the linker (and the few dependencies appimage seems to need) using nix-shell:

$ nix-shell -p zlib.out glibc gcc fuse glib pth  'callPackage ./ld_library_path.nix { }'
# The callPackage is just convenvice to set up NIX_LD_LIBRARY_PATH
$ echo $NIX_LD_LIBRARY_PATH | nix-shorten
/nix/store/hdh*-pth-2.0.7/lib:/nix/store/5v0*-glib-2.50.3/lib:/nix/store/qfj*-libffi-3.2.1/lib:/nix/store/0pw*-libffi-3.2.1-dev/lib:/nix/store/kpa*-zlib-1.2.11-dev/lib:/nix/store/3iv*-glib-2.50.3-dev/lib:/nix/store/acj*-fuse-2.9.7/lib:/nix/store/kjw*-glibc-2.25/lib:/nix/store/sg6*-zlib-1.2.11/lib

$ cat $NIX_CC/nix-support/dynamic-linker
/nix/store/kjwbqnh13dxh6w4pk2gb3ddmhpiaihqg-glibc-2.25/lib/ld-linux-x86-64.so.2
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$NIX_LD_LIBRARY_PATH
$ $(cat $NIX_CC/nix-support/dynamic-linker) ./Subsurface-4.6.2-x86_64.AppImage
./Subsurface-4.6.2-x86_64.AppImage: error while loading shared libraries: ./Subsurface-4.6.2-x86_64.AppImage: ELF file ABI version invalid

No dice, but after a while I realized that the exported LD_LIBRARY_PATH might sneak into the appimage and cause trouble.

So I tried an other approach frequently used to package binaries in nix: (first exit and re-enter nix-shell to make sure the environment is fresh)

`patchelf`

$ cp Subsurface-4.6.2-x86_64.AppImage{,.orig}
$ patchelf \
   --set-interpreter /nix/store/kjwbqnh13dxh6w4pk2gb3ddmhpiaihqg-glibc-2.25/lib/ld-linux-x86-64.so.2 \
   --set-rpath $NIX_LD_LIBRARY_PATH \
   Subsurface-4.6.2-x86_64.AppImage
$ ./Subsurface-4.6.2-x86_64.AppImage
Cannot open /tmp/.mount_tebpjc/.DirIcon

Some progress.. (note: my user can mount other fuse filesystems)

$ exit # exit nix-shell since it alters the environment in other subtle ways
$ ./Subsurface-4.6.2-x86_64.AppImage
Cannot open /tmp/.mount_HeYqBC/.DirIcon
$ gdb ./Subsurface-4.6.2-x86_64.AppImage
(gdb) start
(gdb) break fopen
Breakpoint 2 at 0x7ffff711b910: file iofopen.c, line 97.
(gdb) c
Continuing.

Breakpoint 2, _IO_new_fopen (filename=0x7fffffffb620 "/tmp/.mount_Oa06L2/.DirIcon", mode=0x40562d "rb") at iofopen.c:97
97  iofopen.c: No such file or directory.

# new terminal

$ mount | grep AppImage
Subsurface-4.6.2-x86_64.AppImage on /tmp/.mount_Oa06L2 type fuse.Subsurface-4.6.2-x86_64.AppImage (ro,nosuid,nodev,relatime,user_id=1000,group_id=100)
$ ls /tmp/.mount_Oa06L2
ls: reading directory '/tmp/.mount_Oa06L2': Input/output error
# Guess that could be due to the program being paused in gdb too

while true; do sleep 0.01; ls /tmp/.mount_*; done'ing for fun I sometimes get something like:

 /tmp/.mount_YTKFHc total 136
 drwx------   2 ole  users   4096 sep.   8 03:06 .
 drwxrwxrwt 550 root root  131072 sep.   8 03:06 ..

I can paste LD_DEBUG=all etc, if it might be helpful.

Finally I also tried the Atom appimage with same results.

chroot

Having some OCD tendencies I tried a third approach setting up a chroot environment, but then fuse complained about SUID not being set (which doesn't happen outside the chroot)

Manual mount

Then I found https://github.com/AppImage/AppImageKit/wiki/FUSE and mounted the (unpatched) image manually and ran AppRun manually using the linker:

/nix/store/kj*ihqg-glibc-2.25/lib/ld-linux-x86-64.so.2 ./AppRun
Started in /home/ole/src/nix/tools/foo ...
Moving to /nix/store/kjwbqnh13dxh6w4pk2gb3ddmhpiaihqg-glibc-2.25/lib ...
Error: No .desktop files found

Maybe something with XDG_DATA_DIRS? XDG_DATA_DIRS+=:$(realpath usr/share/) .. same error

strace /nix/store/kj*ihqg-glibc-2.25/lib/ld-linux-x86-64.so.2 ./AppRun 2>&1 | wgetpaste: https://paste.pound-python.org/show/rIzXiKZTGBnncIO6lAnR/

And now I should really sleep :) (after trying to mount in /mnt/ just in case that was important)

probonopd commented 7 years ago

Is there a Live ISO of NixOS that I could try? (I don't feel like installing it into a dedicated partition...)

probonopd commented 7 years ago

Wouldn't it be possible to sudo ln -s /nix/store/kj*ihqg-glibc-2.25/lib/ld-linux-x86-64.so.2 /lib/ld-linux.so.2 or something along those lines? As long as this is not done, NixOS is not gonna be compatible with any third-party Linux applications (which I would say is the whole point of a Linux desktop operating system).

olejorgenb commented 7 years ago

I tried linking (both lib64 and lib linkers) to root. Got further, but still errors:

Stdout: https://paste.pound-python.org/show/qxj5Ino6tg7Y0Fgu2jk0/ Stderr: "Error: Error executing 'subsurface.wrapper'; return code: -1"

(This includes fuse, in LD_LIBRARY_PATH needed to get the actual image to run)

https://nixos.org/nixos/download.html has a livecd

probonopd commented 7 years ago

This is great progress. It means the AppImage is working. :+1: (What is not working is what is inside the AppImage, in this case the subsurface.wrapper script.)

So to summarize, to get AppImage working on NixOS one must link the loader to a location where all "regular" Linux applications expect it to be. I don't think AppImage can do something about this. Apart from that, it appears to be working.

(The wrapper script is not really part of the recent AppImageKit anymore.)

olejorgenb commented 7 years ago

And add fuse and a couple other libraries to LD_LIBRARY_PATH

kevroletin commented 6 years ago

I want to add a link to appimage-run script which runs AppImage executables on nixos (Google sent me here, so the link can be useful for others).

TheAssassin commented 6 years ago

@kevroletin perhaps NixOS could make AppImages launch through that script? Perhaps they want to collaborate with us bringing https://github.com/TheAssassin/AppImageLauncher support to NixOS, which could run that script automatically to run AppImages?

probonopd commented 6 years ago

fwiw, there is a graphical live ISO of NixOS at https://nixos.org/nixos/download.html. :+1: In case someone wants to play with it.

kevroletin commented 6 years ago

@TheAssassin, ah, I didn't realize that:

I was lucky so that AppImage from https://github.com/trufflesuite/ganache worked for me (now I test random images from https://bintray.com/probono/AppImages and I had no luck with 2 images)

I've patched elf to change its default interpreter using these commands:

nix-shell -p binutils stdenv wget dpkg nix-index stdenv.cc
patchelf --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" ganache-1.2.2-x86_64.AppImage

Yep, bringing AppImage support would be cool, but I don't sure who can help. Maybe appimage-run author and contributors can help or can point in the right direction? cc @tilpner @volth @joachifm I'll also ask people in the IRC.

tilpner commented 6 years ago

@kevroletin Try again with appimage-run from recent master (< 4h old, right now)

kevroletin commented 6 years ago

@tilpner, I've checked the latest appimage-run (from Oct 25 10:38:29 2018) and it gave much better results. I've randomly (well, from the first page of https://bintray.com/probono/AppImages) downloaded several images and most of them worked (atom, azpainter, arduino-ide), and one didn't work (audacity, but it's 3 years old, maybe something has changed in a way how AppImages are produced?).

probonopd commented 6 years ago

audacity, but it's 3 years old

Check those newer builds: https://github.com/probonopd/audacity/releases

pinage404 commented 6 years ago

Hi,

I associated AppImage files to appimage-run (i don't know if it's relevant but i'm on KDE)

That added this file ~/.local/share/applications/appimage-run.desktop

#!/usr/bin/env xdg-open
[Desktop Entry]
Exec=appimage-run
MimeType=application/vnd.appimage;
Name=appimage-run
NoDisplay=true
Type=Application

And that added those lines to this file ~/.config/mimeapps.list

 [Added Associations]
+application/vnd.appimage=appimage-run.desktop;
+application/x-iso9660-appimage=appimage-run.desktop;
 # some other stuff here

 [Default Applications]
+application/vnd.appimage=appimage-run.desktop;
+application/x-iso9660-appimage=appimage-run.desktop;
 # some other stuff here

probonopd commented 6 years ago

On other systems, something like appimage-run is not needed. Actually it was one of the design goals of AppImage that in order to run an AppImage, you don't need to install some runtime on the host system first.

Can NixOS do without appimage-run?

pinage404 commented 6 years ago

appimage-run import many dependencies to be able to execute the AppImage Those dependencies are not installed by default NixOS is very minimalist by default

But we could imagine an option that allow AppImage Maybe something like programs.appimage.enable and services.appimage.autoUpgrade.enable

PS: I'm a newbie with NixOS

probonopd commented 6 years ago

Ah, I see, it's a meta dependency package that installs something like the equivalent of the ubuntu-deskop set of packages. In this case, the name is a little misleading, since one could think that it is AppImage specific. Thanks for the clarification.

joepie91 commented 6 years ago

So to summarize, to get AppImage working on NixOS one must link the loader to a location where all "regular" Linux applications expect it to be. I don't think AppImage can do something about this. Apart from that, it appears to be working.

That's not really a realistic solution, though, only a workaround; the entire premise of NixOS is that there isn't a global environment like this, and that all dependencies are explicitly linked to prevent collisions, hence the use of tools like patchelf.

Requiring a system-global link to the loader to run AppImages on NixOS would essentially mean having to give up one of the primary guarantees of NixOS, which defeats the point of using it in the first place.

EDIT: For what it's worth, appimage-run worked for me, at least with FreeCAD.

pinage404 commented 6 years ago

@volth and @reardencode have also contributed to appimage-run on NixOS maybe they can help ?

reardencode commented 6 years ago

I think it's pretty well covered in here -- AppImages will not work on NixOS out of the box due to the fundamental design of the each. NixOS doesn't put any libraries in global scope, and AppImages are specifically designed to run with a bunch of very common libraries in global scope.

Fortunately, appimage-run works great for most AppImages, and it's easy enough to add additional libraries to it if future AppImages expect more dependencies in the global scope. (I had never looked into AppImages nor appimage-run until the day I submitted my one little patch to appimage-run)

probonopd commented 6 years ago

Thanks @reardencode. Are you saying AppImages run well enough on NixOS now so that we can close this issue here? Thanks.

reardencode commented 6 years ago

I'm sure appimage-run will continue to need additional dependencies added from time-to-time. Overall though, AppImages do run well enough on NixOS. Only thing if anything, you might want to add a documentation mention for appimage-run, to help users who come from other operating systems discover how to run AppImages on NixOS.

kamilchm commented 6 years ago

AppImages are specifically designed to run with a bunch of very common libraries in global scope

Is there a list of these common libraries needed?

TheAssassin commented 6 years ago

https://github.com/AppImage/AppImages/blob/master/excludelist

NixOS is very special and different to "average Joe's linux distro"...

kamilchm commented 6 years ago

I got error while loading shared libraries: libatk-bridge-2.0.so.0 when trying to run Beaker Browser >= 0.8.0 and I don't see libatk-bridge on that list. Is that list incomplete or maybe the Beaker Browser package is built incorrectly?

TheAssassin commented 6 years ago

Probably a Beaker Browser issue. You might want to file an issue there. Feel free to highlight me, I'll have a look.

TheAssassin commented 6 years ago

https://github.com/beakerbrowser/beaker/issues/796#issuecomment-442144921

Looks like appimage-run degrades AppImages to self-extracting archives... what a crappy design!

reardencode commented 6 years ago

In order to patch the linker paths, it needs to extract the files and modify them is my understanding.

TheAssassin commented 6 years ago

Still, the least efficient option, and breaks lots of things like AppImageUpdate.

We used to do the same for type 1 support in AppImageLauncher, but now have plans to do the patching needed there in a FUSE filesystem. https://github.com/TheAssassin/AppImageLauncherFS

reardencode commented 6 years ago

Ah, yeah, something like that would be awesome for AppImages on NixOS. Hopefully those who know more than I about NixOS internals are watching that approach!

TheAssassin commented 6 years ago

We only need to patch the AppImage runtime there, though. But using a filesystem is an option for patching linker stuff, etc., too: just build a filesystem that shows the "contents" of the AppImage (i.e., mounts them in a hidden dir, and provides aliases for all files). Then, every binary could be patched on the fly during read() operations.

tilpner commented 6 years ago

@TheAssassin I wrote appimage-run as a quick hack to run a thing I needed, I don't claim it's The Proper Solution™ or anything like that. You're welcome to fix it.

IIRC I extracted them because the FHS wrapper breaks FUSE.

TheAssassin commented 6 years ago

quick hack to run a thing I needed

No offense. I don't claim my old solution for AppImageLauncher is (and hopefully soon: was) a good design.

FHS wrapper

You mean the executable/ELF header (aka runtime)?

breaks FUSE

If you mean the ELF header, you just need to detect the size of it (which is possible using ELF libraries), and then can mount it using the offset option. That's what the runtime internally does, too. If you just want to mount the data as-is, that's the easiest way.

tilpner commented 6 years ago

By FHS wrapper, I mean this.

It is used for Steam support on NixOS, to provide a typical FHS environment to precompiled games. Here, I built a steam-run with the fuse package included, and ran an Inkscape AppImage I got from here:

~ > nix-build --no-out-link -E 'with import <nixpkgs> {}; (steamPackages.steam-chrootenv.override { extraPkgs = p: [ p.fuse ]; }).run'
/nix/store/h0q7pvg8xpiwf1p0icp69hfx7bsiy46p-steam-run
~ > /nix/store/h0q7pvg8xpiwf1p0icp69hfx7bsiy46p-steam-run/bin/steam-run bash
bash-4.4$ Downloads/Inkscape-0.92.3+68.glibc2.15-x86_64.AppImage
fusermount: /nix/store/v6l2sacryfr88yqq0pq7sia8wfgm9q31-wrapper.c:203: main: Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())' failed.

Cannot mount AppImage, please check your FUSE setup.
You might still be able to extract the contents of this AppImage
if you run it with the --appimage-extract option.
See https://github.com/AppImage/AppImageKit/wiki/FUSE
for more information
open dir error: No such file or directory
bash-4.4$

The wrapper.c mentioned in the error is the SUID wrapper NixOS uses because /nix/store can't have SUID executables.

There are other options, I tend to use bubblewrap instead of this wrapper (though I'm not sure it would help in this case), but that would have been more tedious to merge into nixpkgs.

probonopd commented 6 years ago

In order to patch the linker paths

AppImages imho should be treated as monolithic objects, not to be patched. The proper solution is to install a linker in the "usual place" so that NixOS gains compatibility with "normal" binaries. Possibly this can be done in a private namespace so that it only affects those binaries that need it.

tilpner commented 6 years ago

@probonopd It's not just the linker, it also needs a ton of libraries. And that's sort-of what appimage-run does, use namespaces to provide a filesystem that looks like what the AppImage expects, including linker and libraries at the usual places.

TheAssassin commented 6 years ago

It seems more and more that NixOS never wanted to run arbitrary binaries and/or FUSE...

probonopd commented 6 years ago

It seems more and more that NixOS never wanted to run arbitrary binaries

That is what I have my main "philosophical" issue with. If the NixOS developers think that every single application shall be specifically ported/modified to run on their particular OS distribution then, well, good luck with that. Then NixOS behaves like it is its own operating system with a tiny, tiny market share, rather than just another Linux distribution. How many application developers are going to care?

tilpner commented 6 years ago

That (upstream maintaining a package in nixpkgs) is not an expectation I have observed in the community. FOSS applications are often easy to package, people just add things they want to use themselves, and there are a few (admittedly less pretty) solutions to get pre-compiled software to run.

reardencode commented 6 years ago

That is what I have my main "philosophical" issue with. If the NixOS developers think that every single application shall be specifically ported/modified to run on their particular OS distribution then, well, good luck with that. Then NixOS behaves like it is its own operating system with a tiny, tiny market share, rather than just another Linux distribution. How many application developers are going to care?

There are standard patching techniques that can be applied to get any binary to run, and that's what appimage-run does. The real problem is that so many applications, including AppImage assume far too much about *nix systems just because most GNU/Linux distributions do things the same. The same types of problems often arise when porting to the BSDs.

tilpner commented 6 years ago

@reardencode appimage-run doesn't really patch anything

probonopd commented 6 years ago

We should really better get our act together to make all desktop Linux distributions to actually deliver on those assumptions (i.e., have less unnecessary differences), but that is just my 2 cents.

That (upstream maintaining a package in nixpkgs) is not an expectation I have observed in the community. FOSS applications are often easy to package, people just add things they want to use themselves

This approach works best for people who are

Not interested in closed-source applications (not that I prefer them but sometimes there are reasons to run them, too)
Happy with modifications done by "random" third parties (rather than the original application author); resulting with a derivative that the original author in some cases may not be able to support; and/or
Happy to develop (rather than: just download and run) things

(Yes, I know that my statements are controversial but that's how one can look at things, too.)

reardencode commented 6 years ago

@reardencode appimage-run doesn't really patch anything

Doesn't it apply the patchelf changes to get linkage to work?

Ekleog commented 6 years ago

(disclaimer: I don't use AppImages myself, but came upon here from the NixOS repo)

@TheAssassin FWIW, NixOS runs precompiled binaries and FUSE quite well. However, it sounds like (didn't test) it can't do both at the same time due to limitations of the Linux kernel (that from my reading are “no FUSE in a chroot in a mountns”, but again I didn't test). Which, I hope you'll agree with me, is a rather infrequent use case… maybe as infrequent as NixOS itself?

(also, actually there are usually ways to run proprietary applications without the chroot-in-a-mountns hack, using patchelf, except for the worst ones like Steam -- it sounds like AppImages are currently in that last category)

Now, back to the actual topic: if AppImage can use environment variables to figure out where the linker is located for the in-image executables, then I think they will be possible to execute correctly on NixOS without the chroot-in-mountns hack. For the main executable that runs the whole image, we can just patchelf on it and be good with that.

(Disclaimer bis: I am not familiar with the way AppImages work, so may be wrong with this. The idea is that we need a way to override all hardcoded paths to system utilities if we want to remove the chroot-in-mountns hack.)

kamilchm commented 6 years ago

I'm not an expert on Linux and most stuff you guys discussing here but I will be thankfull if you can help me with one basic use case: I want to run Beaker Browser AppImage on NixOS. I see 3 actors here:

The state is: error while loading shared libraries: libatk-bridge-2.0.so.0

I'm willing to make changes in places where I can do a PR but it's not clear to me if:

Beaker AppImage is fine not having libatk-bridge inside and so there's missing libatk-bridge first in the excludelist and then in the appimage-run.
Beaker AppImage is broken not having libatk-bridge so both exludelist and appimage-run are fine but there's a bug in the Beaker Browser packaging.

This is my simple inexperienced user's view of my use case.

probonopd commented 6 years ago

I'd say it's not a "bug" anywhere: Most mainstream desktop distributions apparently happen to ship libatk-bridge-2.0.so.0 as part of the default desktop installation whereas NixOS happens not to do that.

Since many AppImages are built against and tested with "most mainstream desktop distributions", it cannot be expected to work on NixOS unless NixOS, in its default installation, ships the same "least common denominator" set of libraries that "most mainstream desktop distributions" ship in theirs.

So, the "right thing to do" would be:

NixOS should define what their desktop default installation contains
Ideally, NixOS should provide a Live ISO with the desktop default installation so that application authors could easily test their applications on it
The AppImage project might consider the NixOS desktop default installation when putting together the excludelist. Note that the excludelist lists libraries that we think must not be bundled with each application, as it is assumed that bundling those is considered harmful and the system-provided versions of those libraries should be used. This does not mean that if a system has all libraries on the excludelist installed locally, then all AppImages will run. Since the application authors may have assumed other libraries (which are not on the excludelist because they are not harmful to bundle) to also be "there" on each target system (most likely, based on evidence).

Ekleog commented 6 years ago

Doesn't that contradict the idea that AppImages are “Linux apps that run anywhere” (wording from the front page)?

For AppImages to be Linux apps that run anywhere, the list would need to be a non-overridable excludelist for which all other libraries would need to be bundled, so that every platform that wants to support AppImages know it only needs to provide the libraries listed in this excludelist.

probonopd commented 6 years ago

True.

So far we have been working from the assumption that we cannot influence what the "Leading Linux distributions" happen to install by default, and just take that set for granted.

Since this may slowly be changing (with some distributions specifically geared toward running AppImages) it might indeed be worthwhile to do what you are suggesting.

wucke13 commented 4 years ago

I have a probably dumb question: Why aren't AppImages statically linked binaries (that run on anything linux, even NixOS, without needing any shared object file on a specific place)?

The whole point about application portability in the linux world is about where do our applications expect which version of what shared object. So to me it seems to be a poor choice to use a dynamically linked binary to solve this problem about portability, as it again introduces assumptions about the target system.

probonopd commented 4 years ago

I have a probably dumb question: Why aren't AppImages statically linked binaries (that run on anything linux, even NixOS, without needing any shared object file on a specific place)?

To make them really static, two things would need to be static:

The AppImage runtime. (There is a proof-of-concept that shows that this is doable.)
Everything that goes into the AppImage. (Alternatively one could bundle a private ld-linux, glibc, and everything else. There is a proof-of-concept that shows that this is doable.)

This would increase the AppImage size and some people might think this would be "bloated".

wucke13 commented 4 years ago

Everything that goes into the AppImage.

That is not going to happen, unfortunately. Too many things cannot be built statically without a lot of effort (and almost no upstream project is willing to put that effort). Just shipping all the shared object files would be the saner variant here IMO.

people might think this would be "bloated".

Yes! That is the whole point of distro unspecific images of software. Shipping everything needed to run. If you want efficient unbloated packaging, go treat your distro's maintainers.

AppImage / AppImageKit