Consider setting RPATH for executables and libraries

[mikix]

The core snaps should probably set an RPATH for executables and libraries to help when used in classic snaps. Otherwise, there is a risk of symbol incompatibility.

Checking RPATH Currently, this command gives no output:

$ readelf -d /snap/core18/current/usr/bin/python3 | grep RPATH

But ideally, something like the following would happen:

$ readelf -d /snap/core18/current/usr/bin/python3 | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [/snap/core18/current/lib/x86_64-linux-gnu]

I don't know if this would cause problems when in a properly confined snap. But maybe an RPATH that checked /snap/core18 first then checked the system locations would work for both?

Example of problem I recently encountered this issue in my deja-dup snap. It's currently built as a classic snap based off core18.

The problem occurred when running a python3 command that loaded a compiled module provided by my snap.

Snapcraft correctly set an RPATH for me on my snap's executables and libraries. But core18's python3 interpreter did not.

So when this compiled python module was imported, (the wrong) libc6 was already loaded by the interpreter, and the import raised this error because librt from core18 was confronted with libc6 from my host system (Ubuntu 20.04):

/snap/core18/current/lib/x86_64-linux-gnu/librt.so.1: undefined symbol: __clock_nanosleep, version GLIBC_PRIVATE

My workaround was just to ship a version of python3 in my own snap and use that (since snapcraft then sets an RPATH for that executable and everything is fine).

(https://bugs.launchpad.net/deja-dup/+bug/1875215)

[cjp256]

This may be a bigger issue for core20 classic snaps, because snapcraft is more aggressive about not staging anything that is found in the base's manifest, unless specifically listed [in stage-packages]. It is something we can work around of course, but this was not taken into consideration (at least I had not). @sergiusens

FWIW, if it is decided that setting RPATH is appropriate on the base, I'd suggest considering doing it on a new, classic-version of the base to prevent impacting strict snaps.

[anonymouse64]

I think this may break strictly confined snaps, but if there is a way that this can be done without breaking strictly confined snaps, I think it is still probably an ABI change and as such should not be done for core and core18, but may be doable for the core20 snap if we can prove that strict snaps are not broken with this and merge things quickly.

I just checked and I don't see any binaries that set DT_RPATH in the core18 or core20 snaps at all, however numerous binaries set DT_RUNPATH which is the newer version of DT_RPATH and supports being set to a list of directories to search. So what would probably work is to set DT_RUNPATH to something like [/snap/core20/current/lib/x86_64-linux-gnu, /lib/x86_64-linux-gnu] although this would incur spurious denials for strict that would be quite frequent if not silenced with a deny rule in AppArmor.

@cjp256 To my knowledge, I don't think we have ever discussed the concept of having different bases for different confinement models, so that's would require a lot of design work to implement.

CC @jdstrand @xnox @mvo5

[anonymouse64]

Hmm, actually I wonder if setting /snap/core20/current will open up a new can of worms for classic snaps in that would expose them to the base getting refreshed underneath them? Maybe they already have this can of worms because they're classic and not isolated from the host at all?

/me doesn't remember how base snap updates work w/respect to running applications

[ogra1]

don't forget that the core snap is also a root filesystem for Ubuntu Core ... not sure what the actual implications here are but i imagine hardcoding rpaths could cause some issues ...

[xnox]

Hmm, actually I wonder if setting /snap/core20/current will open up a new can of worms for classic snaps in that would expose them to the base getting refreshed underneath them? Maybe they already have this can of worms because they're classic and not isolated from the host at all?

/me doesn't remember how base snap updates work w/respect to running applications

No different to how dpkg replaces libc6 underneath running apps during classic upgrades. All libraries are loaded and memory-mapped, one can even inspect and find currently running binaries that reference deleted files. So i'm not too worried about refreshes deleting base snaps.

I would be more comfortable with runpath to be set to the current revisions the app is launched with, i.e. [/snap/core20/2345/lib/x86_64-linux-gnu] which would need to be snapd managed.

That is because /current/ is a moving target that can go forwards, and backwards with rollback. And we only have forward ABI guarantee, not backwards one. And a restart of running processes might be required.

Also I don't believe it is safe for classic snaps to load random libraries from the host. They should only load libraries from within their own snap, and base snap, and not the host. Because we really cannot tell at all if libc6 on the host is at all good enough better/worse than the one in the app or base snaps.

[xnox]

don't forget that the core snap is also a root filesystem for Ubuntu Core ... not sure what the actual implications here are but i imagine hardcoding rpaths could cause some issues ...

Doesn't /snap/core18/current exist on either classic or core? (irrespective of which revision of core18 was used as base rootfs for a particular boot of core)

[anonymouse64]

Doesn't /snap/core18/current exist on either classic or core?

Yes, it exists, but strict snaps cannot access the base at this path, since they will have pivot_root/chroot'd so that what's in /snap/core18/current is now at / for them.

[anonymouse64]

I would be more comfortable with runpath to be set to the current revisions the app is launched with, i.e. [/snap/core20/2345/lib/x86_64-linux-gnu] which would need to be snapd managed.

How could snapd set the rpath for a revision in the core20 snap given that the files in the core20 snap are read-only?

[xnox]

We can use "Rpath token expansion" in RUNPATH to encode things relative the file location. Thus all binaries inside core snap will encode RUNPATH ${ORIGIN}/../{LIB} or some such, which ld.so will correctly resolve, chrooted to base or not.

[anonymouse64]

That sounds like it would work the best actually as that would do the right thing for either classic snaps or strict snaps. The only question I would have then is how do we go about automating setting that because I assume this will be a hook we use the core* snap build scripts.

[cmatsuoka]

Note that setting the runpath would be best done at binary link time since patchelf is known to fail in some cases.

[sergiusens]

What @xnox describes about using ${ORIGIN} is what snapcraft does today for classic for any given snap. This would be best done at link time as @cmatsuoka mentions, so more work than expected as it affects all of Ubuntu (for the case of core20). To answer @cjp256, I did consider core20 and snapcraft, you need to explicitly stage or provide the python3 runtime for things to work (no magic) and if the base ever supports this, you just stop staging it.

[xnox]

@sergiusens can you point me at the ${ORIGIN} stuff that snapcraft already does? Cause i think we need to do that, against core20 binaries at build time. Cause core20 is dual-purpose, can be rootfs, can be confined strict, can be classic.

[sergiusens]

On Mon, 2020-05-11 at 15:20:27 -0300, Dimitri John Ledkov wrote:

@sergiusens can you point me at the ${ORIGIN} stuff that snapcraft already does? Cause i think we need to do that, against core20 binaries at build time. Cause core20 is dual-purpose, can be rootfs, can be confined strict, can be classic.

We basically run ldd and then patchelf https://github.com/snapcore/snapcraft/blob/master//snapcraft/internal/elf.py#L550 Library paths on the base are absolute, in-snap ones use ${ORIGIN}.

What do you mean by "at build time"?

[xnox]

Just for myself, checking notes I see that python paths are otherwise dynamic and "relocatable"

/snap/core20/current/usr/bin/python3.8 -c 'import sys; print(sys.path)'
['', '/snap/core20/current/usr/lib/python38.zip', '/snap/core20/current/usr/lib/python3.8', '/snap/core20/current/usr/lib/python3.8/lib-dynload', '/home/xnox/.local/lib/python3.8/site-packages', '/snap/core20/current/usr/lib/python3/dist-packages']

[xnox]

Hm....

snap run --shell ubuntu-image
$ env | grep -e snap -e SNAP
SNAP_REVISION=166
SNAP_USER_COMMON=/home/xnox/snap/ubuntu-image/common
SNAP_INSTANCE_KEY=
SNAP_CONTEXT=dFq95s49S4D2J6fmg1dHFlXsPQJNHRXFH3CfS10xNRGS
SNAP_ARCH=amd64
SNAP_INSTANCE_NAME=ubuntu-image
SNAP_USER_DATA=/home/xnox/snap/ubuntu-image/166
SNAP_REEXEC=
PYTHONPATH=/snap/ubuntu-image/166/lib/python3/site-packages:
SNAP=/snap/ubuntu-image/166
SNAP_COMMON=/var/snap/ubuntu-image/common
SNAP_VERSION=1.9+snap2
SNAP_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void
SNAP_COOKIE=dFq95s49S4D2J6fmg1dHFlXsPQJNHRXFH3CfS10xNRGS
SNAP_DATA=/var/snap/ubuntu-image/166
LD_LIBRARY_PATH=/snap/ubuntu-image/166/usr/lib:/snap/ubuntu-image/166/lib:/snap/ubuntu-image/166/usr/lib/lib-arch:/snap/ubuntu-image/166/lib/lib-arch
XDG_RUNTIME_DIR=/run/user/1000/snap.ubuntu-image
SNAP_NAME=ubuntu-image
XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop:/var/lib/snapd/desktop
PATH=/snap/ubuntu-image/166/usr/bin:/snap/ubuntu-image/166/bin:/snap/ubuntu-image/166/sbin:/home/xnox/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/xnox/canonical/ubuntu-cdimage/bin:/home/xnox/canonical/ubuntu-cdimage/bin

Why is there no indication as to what is my base? Why base is not in LD_LIBRARY_PATH nor PATH at all? Why is there no SNAP_BASE=/snap/core18/123 or some such?

Also, I somewhat expected for a new mount namespace to be started with, with my default interpreper to be bind-mounted for me. I.e. /snap/core18/lib64/ld-linux-x86-64.so.2 to be bind mounted onto /lib64/ld-linux-x86-64.so.2.

[anonymouse64]

Also, I somewhat expected for a new mount namespace to be started with

We don't create mount namespaces for classic snaps by design thus far, only if you do parallel installs of a classic snap will we enter a new mount namespace for the classic snap.

with my default interpreper to be bind-mounted for me. I.e. /snap/core18/lib64/ld-linux-x86-64.so.2 to be bind mounted onto /lib64/ld-linux-x86-64.so.2.

Why did you expect this? Classic snaps are meant to run "on the host" so if we did that host things would break, i.e. when an editor goes to run my compiler as a deb, the compiler shouldn't be looking at the interpreter from the base snap?

I really interpret this request/issue as a request to make things that run from inside a classic snap do the right thing more often, such that it's easier to build classic snaps with less hacks, not that we should make classic snaps be more like strict snaps.

[xnox]

Also, I somewhat expected for a new mount namespace to be started with

We don't create mount namespaces for classic snaps by design thus far, only if you do parallel installs of a classic snap will we enter a new mount namespace for the classic snap.

with my default interpreper to be bind-mounted for me. I.e. /snap/core18/lib64/ld-linux-x86-64.so.2 to be bind mounted onto /lib64/ld-linux-x86-64.so.2.

Why did you expect this? Classic snaps are meant to run "on the host" so if we did that host things would break, i.e. when an editor goes to run my compiler as a deb, the compiler shouldn't be looking at the interpreter from the base snap?

Hm, true. VScode should totally call /usr/bin/gcc with host's ld.so. or even just whatever pops up first in the path as gcc.

I really interpret this request/issue as a request to make things that run from inside a classic snap do the right thing more often, such that it's easier to build classic snaps with less hacks, not that we should make classic snaps be more like strict snaps.

I guess, my question is for things like ssh client. Should that be called from host or base snap? especially since the focal's host /etc/ssh/ssh_config is incompatible with ssh from core/core18 snaps.

[anonymouse64]

I guess, my question is for things like ssh client. Should that be called from host or base snap? especially since the focal's host /etc/ssh/ssh_config is incompatible with ssh from core/core18 snaps.

Ideally we probably want ssh to be staged in your snap and you should use the one from /snap/my-classic-snap/current/usr/bin/ssh, with some special sauce configured to use the configuration from your snap or the base snap if it's in the base snap, but if that's difficult/not possible, then probably the next best bet is to use the host's version from $PATH and not use the one from the base due to the problem you raise specifically.

[sergiusens]

On Tue, 2020-05-12 at 12:50:34 -0300, Dimitri John Ledkov wrote:

Just for myself, checking notes I see that python paths are otherwise dynamic and "relocatable"

Yeah, this is then entire premise on how virtual envs work, which we take complete advantage of for the python plugin for core20