Union/overlay "filesystem" using LD_PRELOAD

[probonopd]

We experimented with this for klik 9 years ago: https://www.winehq.org/pipermail/wine-devel/2007-November/060627.html

Someone else seems to have had some success with this idea: http://algoholic.eu/unionfs_by_intercept/

deb2snap seems to be going down that path, too: https://github.com/mikix/deb2snap/blob/master/src/preload.c Actually the whole deb2snap thing could be massaged into a deb2appimage thing relatively easily, it seems. Discussion: https://lists.ubuntu.com/archives/snappy-devel/2015-February/000282.html

user-union: http://www.dwheeler.com/user-union/ (thanks for the hint @stsp)

Or we could use FUSE: https://github.com/rpodgorny/unionfs-fuse

CARE/PRoot: https://packages.debian.org/unstable/main/care "PRoot is basically better fakeroot-ng plus fakechroot, or schroot minus root privilege" http://www.slideshare.net/cvinc02/proot-improvedkernelcompat https://github.com/proot-me/PRoot Unmaintained?

cde: cde ls creates an empty cde-package directory where I can add (not overwrite!) an AppDir, then run it with cde-exec appname. It does an overlay/union

[stsp]

I don't know, but I'd be interested in the results. Lets continue this discussion in probonopd/AppImageKit#267

Just a quick look:

unionfs_by_intercept seems to be only intercepting syscalls but not library calls. The problem with this is that glibc calls the syscalls internally, you can't intercept those. So when you call fopen(), you can't count on intercepting open() - it won't work. You need to intercept all library calls too.

preload.c is definitely interesting, but, being just a part of other project, I wonder how generic/configurable is it. I haven't tried it.

https://wiki.debian.org/FakeRoot is also an interesting project in that regard, but it also is targeted for the very specific use case.

unionfs-fuse is absolutely great, but IIRC it starts a server process per every mount point, and slows down things considerably (I mean, really-really considerably).

There was also a ptrace-based FS redirector somewhere. Have to google for it a bit more.

user-union (which I co-authored) was the most feature-rich and fast at time, but now I checked and it doesn't even build easily. :( I'll try to find some time to get it fixed.

[stsp]

So when I was dealing with this (and that was long ago) I was combining unionfs-fuse for non performance-critical parts, with user-union for performance-critical parts. But many other projects seems to have emerged since then...

[stsp]

http://www.dwheeler.com/user-union/ (thanks for the hint @stsp)

The interesting thing is that you already had it in your repository even before my hint. I don't know if you succeeded with building it (the problem seems to depend on glibc version), but if you didn't, you need to do: ./configure --disable-private-libc-namespace

Then it should build and work.

[probonopd]

Thanks. can you give your opinion on preload.c vs. user-union, do you see clear upsides/downsides of one over the other?

[stsp]

Unfortunately I don't see a list of features of preload.c (like user-union's man page that I suggest you to read for a basic idea about the feature set), and I have not tried it myself, so I can only very basically judge on the code look. I wonder if preload.c is about FS overlaying. For example I can see the FS redirection code there, but I can't see the FS overlay code. So if I am not mistaken, the goal of preload.c is just a very small subset of what user-union can do. With preload.c you can create the private sandbox, which is great. You can do the same with fakechroot too. But with user-union you can do things like unionfs-fuse does: mount multiple directories into a single mount-point, one on top of another in a stack. If you don't need that functionality, then user-union and unionfs-fuse are not your friends, but if you need that, then it is exactly their task.

[stsp]

So if we are talking only about the FS redirection and not union mounts, then it is entirely possible that preload.c does things well. It looks fairly complete in that regard, lots of syscalls and library calls wrapped, maybe as much as user-union does, maybe even more. It would be hard to name the winner here unless you need something what user-union is really designed for.

[stsp]

Well, as long as this ticket is called "Union/overlay "filesystem" using LD_PRELOAD" and you already experimented with preload.c, my question is: does it really do overlays for you? I don't see such code in there!

[probonopd]

Kinda: I can use it to "overlay" the contents of the AppImage over / with the result that if I launch $APPDIR/usr/bin/foo and it calls /usr/bin/bar, then bar works. But strangely it seems to be half-implemented because a directory listing or file open dialog box for /usr/bin will ONLY show the contents of $APPDIR/usr/bin at /usr/bin but NOT in combination with the real /usr/bin... bug or feature or just not implemented?

[stsp]

Kinda: I can use it to "overlay" the contents of the AppImage over / with the result that if I launch $APPDIR/usr/bin/foo and it calls /usr/bin/bar, then bar works.

What is "then bar works"? Does $APPDIR/usr/bin/bar work or /usr/bin/bar works? A big difference.

[probonopd]

/usr/bin/bar works; clarified my comment above.

[stsp]

/usr/bin/bar works; clarified my comment above.

Is this really what you want? With user-union you can get:

1. Attempt to run $APPDIR/usr/bin/bar
2. If 1 does not exist, the lower layer is tried, so /usr/bin/bar is started.

If it doesn't run $APPDIR/usr/bin/bar then there is no overlay at all (as the code suggests anyway)

[probonopd]

I think I like some aspects of user-union better, but it is much more complicated to use. What is great about preload.c is that a) it is just one file, b) you export one LD_PRELOAD variable and one UNION_PRELOAD variable and be done with it. In contrast, user-union comes with a convoluted bash wrapper script.

Maybe you can help me to make it as easy?

[stsp]

So what kind of overlay is this? Am I right that you mount $APPDIR to / and if progs execute /usr/bin/bar, the look-up in $APPDIR overlay is not done, and /usr/bin/bar is executed as is?

[probonopd]

I would like $APPDIR to be overlayed to /

• If both /usr/bin/foo and $APPDIR/usr/bin/foo exist and /usr/bin/foo is accessed, then it should be used from $APPDIR/usr/bin/foo
• If only $APPDIR/usr/bin/foo exists and /usr/bin/foo is accessed, then it should be used from $APPDIR/usr/bin/foo (edit: fixed typo)
• If only /usr/bin/foo exists and /usr/bin/foo is accessed, then it should be used from /usr/bin/foo

Special directories like /home, /dev, /proc etc. should be left untouched. Writes should never go to the $APPDIR.

[stsp]

I would like $APPDIR to be overlayed to /

user-union supports mount points. I.e. you can overlay $APPDIR with / and put the resulting mount to /mnt/overlay.

If only $APPDIR/usr/bin/foo exists and /usr/bin/foo is accessed, then it should be used from /usr/bin/foo

Are you sure? I think $APPDIR/usr/bin/foo should be used in this case.

What is great about preload.c is that you export one LD_PRELOAD variable and one UNION_PRELOAD variable and be done with it.

That's because user-union supports a very complex mounts. For example, you can have any amount of R/O underlays with 1 or 0 overlay over them. Also you can have multiple mounts at once. Such flexibility requires setting many env vars unfortunately, so it was simpler to add the script.

half-implemented because a directory listing or file open dialog box for /usr/bin will ONLY show the contents of $APPDIR/usr/bin but NOT in combination with the real /usr/bin... bug or feature or just not implemented?

Seems not implemented, same in user-union. :) File listing is a quite difficult to implement feature.

[probonopd]

Are you sure? I think $APPDIR/usr/bin/foo should be used in this case.

You are right, I corrected my typo above.

Yes, I'm sure user-union is much more powerful, maybe too powerful for our simple usecase? But maybe we could do a simplified subset of the functionality, with virtually no runtime options? Actually that would be best, just set LD_PRELOAD and the rest is all hardcoded in the lib.

[stsp]

So I guess if you have just one mount with one overlay and one underlay, and the mount-point is equal to the underlay path, then user-union is very unlikely to offer anything over preload.c. This is what you can do with user-union -a $APPDIR /

To avoid the script, you can do the trick: user-union -a $APPDIR / -n

-n will just print the needed env vars and exit. You can set those vars yourself and use LD_PRELOAD then.

[probonopd]

What do you think about https://github.com/proot-me/PRoot? http://www.slideshare.net/cvinc02/proot-improvedkernelcompat https://www.reddit.com/r/linux/comments/1s7jgj/care_automatically_create_portable_reproducible/

[stsp]

If you have just the basic requirements and preload.c is enough, why to search for more?

What do you think about https://github.com/proot-me/PRoot?

The description says: chroot, mount --bind, and binfmt_misc without privilege/setup

Doesn't sound like it supports overlay mounts. I haven't really looked into it.

[probonopd]

If you have just the basic requirements and preload.c is enough, why to search for more?

I just like to understand the options we have :-)

[stsp]

Then you can try the -n trick of user-union to get the needed vars printed, recreate them and compare to preload.c.

[probonopd]

Found an issue with preload.c:

This recipe does not work correctly with union: true but does with binpatch: true:

app: OpenSCAD
binpatch: true

ingredients:
  packages:
    - openscad-nightly
  dist: trusty
  sources: 
    - deb http://archive.ubuntu.com/ubuntu/ trusty main
    - deb http://download.opensuse.org/repositories/home:/t-paul/xUbuntu_14.04/ ./

script:
  - ls

The difference is that with binpatch: true the examples are shown in this window correctly:

With union: true the examples are not shown and we get Error reading examples.json: examples.json: cannot open file.

---

Unlike with binpatch: true, it also does not work correctly with this recipe: https://github.com/probonopd/AppImages/blob/master/recipes/meta/OBS-Studio.yml

---

Nor does it work with this recipe: https://github.com/probonopd/AppImages/blob/master/recipes/meta/Mpv.yml Here, Samba has .so libraries in a subdirectory of lib/ and if I move them out from there, they are no longer found and the system ones are loaded instead...

[TheAssassin]

We should try to find out why. Binpatch is pretty much a hack, using the union file system is cleaner and we don't have to apply this cd .../usr hack.

[probonopd]

Also check https://github.com/Barthalion/hai which uses union-fuse and fakechroot.