rebase: local-repo rebase fails due to SELinux denial

[lucab]

Host system details

State: idle
Deployments:
* ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 32.20200726.3.1 (2020-08-12T05:29:32Z)
                    Commit: 2579b41aa614c3a40b9e24ff0b9dd288f99222dc3ed3a527ef0d8e8667196ff5
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

Expected vs actual behavior

I have an FCOS ostree-repo-in-squashfs which I'm trying to use as the source for a local rebase. However, SELinux is preventing rpm-ostree rebase from using it, and I'm unsure what needs to be adjusted (the context of the mount vs dbus-related logic).

$ cat /mnt/refs/heads/fedora/x86_64/coreos/stable 
902c88022e834fb7998140734fae01b75f78b80f63364e9c5309d62d9a260b7c

$ findmnt /mnt
TARGET   SOURCE     FSTYPE   OPTIONS
/var/mnt /dev/loop0 squashfs ro,relatime,seclabel

$ ls -lZd /mnt/
drwxr-sr-x. 7 root root system_u:object_r:unlabeled_t:s0 123 Aug 24 16:30 /mnt/

$ sudo rpm-ostree rebase '/mnt:fedora/x86_64/coreos/stable'
error: The connection is closed

The following denial is logged to the journal:

Aug 28 11:00:19 localhost rpm-ostree[2306]: client(id:cli dbus:1.28 unit:session-3.scope uid:0) added; new total=1
Aug 28 11:00:19 localhost audit[859]: AVC avc:  denied  { read } for  pid=859 comm="dbus-broker" path="/var/mnt" dev="loop0" ino=20313 scontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=0                                                                                                                                                                   
Aug 28 11:00:19 localhost kernel: audit: type=1400 audit(1598612419.645:260): avc:  denied  { read } for  pid=859 comm="dbus-broker" path="/var/mnt" dev="loop0" ino=20313 scontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=0
Aug 28 11:00:19 localhost sudo[2301]: pam_unix(sudo:session): session closed for user root
Aug 28 11:00:19 localhost audit[859]: SYSCALL arch=c000003e syscall=47 success=yes exit=503 a0=d a1=7ffc1f63ee40 a2=40000040 a3=ffffffff items=0 ppid=858 pid=859 auid=4294967295 uid=81 gid=81 euid=81 suid=81 fsuid=81 egid=81 sgid=81 fsgid=81 tty=(none) ses=4294967295 comm="dbus-broker" exe="/usr/bin/dbus-broker" subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 key=(null)                                           
Aug 28 11:00:19 localhost kernel: audit: type=1300 audit(1598612419.645:260): arch=c000003e syscall=47 success=yes exit=503 a0=d a1=7ffc1f63ee40 a2=40000040 a3=ffffffff items=0 ppid=858 pid=859 auid=4294967295 uid=81 gid=81 euid=81 suid=81 fsuid=81 egid=81 sgid=81 fsgid=81 tty=(none) ses=4294967295 comm="dbus-broker" exe="/usr/bin/dbus-broker" subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 key=(null)
Aug 28 11:00:19 localhost rpm-ostree[2306]: client(id:cli dbus:1.28 unit:session-3.scope uid:0) vanished; remaining=0
Aug 28 11:00:19 localhost rpm-ostree[2306]: In idle state; will auto-exit in 61 seconds

I'm not sure why dbus-broker is trying to access the repo directory (as opposed to rpm-ostreed directly), and what the correct context for the resource in this operation would be.

[cgwalters]

Anything that is unlabeled_t is likely to cause issues like this. You can use mount -o context= to set a context.

Now the specific reason a denial is happening is because the file descriptor is passed from client → dbus-broker → rpm-ostreed. The first is unconfined_t which can apparently interact with unlabeled_t, but dbus_broker_t can't.

We could possibly mitigate this if we passed the descriptor directly from client → rpm-ostreed (using a private socket) but it's still likely that install_t (the context of rpm-ostreed) can't interact with unlabeled_t either.

So basically I'd use mount -o context=.

[lucab]

Thanks for the insights! So, I was missing the fact that SELinux does re-check permissions when passing an already-open FD. Thus it is now my understanding that the original resource context must be allowed for both the intermediate proxy (dbus-brocker) and the final receiver (rpm-ostreed).

It looks like an install_t resource type could work here. There is an additional gotcha that -o rootcontext should be used instead of -o context, otherwise the relabeling is also blocked:

Aug 28 12:36:15 localhost sudo[1824]: core : TTY=pts/0 ; PWD=/var/home/core ; USER=root ; COMMAND=/usr/bin/mount -t squashfs -o context=system_u:object_r:install_t:s0 --source ostree.img --target /mnt/
Aug 28 12:36:15 localhost audit[1826]: AVC avc:  denied  { relabelto } for  pid=1826 comm="mount" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:object_r:install_t:s0 tclass=filesystem permissive=0      

Other than that, it seems to work fine:

[core@localhost ~]$ sudo mount -o rootcontext="system_u:object_r:install_t:s0" --source ostree.img --target /mnt/
[core@localhost ~]$ sudo rpm-ostree rebase '/mnt:fedora/x86_64/coreos/stable'
Writing objects: 1... done
Staging deployment... done
...
Run "systemctl reboot" to start a reboot

Another minor thing is that daemon seems to hold the FD even after returning to the client, until it shuts down after idleing:

...
Run "systemctl reboot" to start a reboot

$ sudo lsof +D /mnt/
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
rpm-ostre 4912 root   16r   DIR    7,0      123 20313 /var/mnt

$ sudo umount /mnt/
umount: /mnt/: target is busy.

$ sleep 70

$ sudo umount /mnt/
# OK

[cgwalters]

Another minor thing is that daemon seems to hold the FD even after returning to the client, until it shuts down after idleing:

https://github.com/coreos/rpm-ostree/pull/2211

[jlebon]

Yeah, definitely hit this before when trying to rebase directly from an extracted FCOS OSTree commit tarball. It's not just when using unlabeled things. E.g. testing this now

# mkdir /srv/fcos && cd /srv/fcos
# tar -xf /var/tmp/fedora-coreos-32.20200827.20.0-ostree.x86_64.tar
# rpm-ostree rebase /srv/fcos:fedora/x86_64/coreos/testing-devel
error: The connection is closed
# journalctl -e --no-pager --grep denied
Aug 28 13:42:01 cosa-devsh audit[922]: AVC avc:  denied  { read } for  pid=922 comm="dbus-broker" path="/var/srv/fcos" dev="vda4" ino=11116496 scontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_t:s0 tclass=dir permissive=0
Aug 28 13:42:01 cosa-devsh kernel: audit: type=1400 audit(1598622121.460:271): avc:  denied  { read } for  pid=922 comm="dbus-broker" path="/var/srv/fcos" dev="vda4" ino=11116496 scontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_t:s0 tclass=dir permissive=0

So here the denial is between system_dbusd_t and var_t.

We can fix this using chcon -t var_run_t -R .. This comes up in the MCO too, see https://github.com/openshift/machine-config-operator/pull/1979#issuecomment-669008742 and https://github.com/openshift/machine-config-operator/blob/36f37f2d6009affe8174854f5ef5538e0cc49034/pkg/daemon/update.go#L262-L268.

We should probably try to broaden the policy. That'll avoid OCP/OKD having to do a mass relabel each time.

[lucab]

One interesting nit here is that only the top-level directory context mattered in the unlabeled case. That is, the following mounted repository worked fine (result of the -o rootcontext mount above):

$ ls -laZ /mnt/
total 5
drwxr-sr-x.   7 root root system_u:object_r:install_t:s0    123 Aug 24 16:30 .
drwxr-xr-x.  23 root root system_u:object_r:var_t:s0       4096 Aug 28 13:25 ..
-rw-r-----.   1 root root system_u:object_r:unlabeled_t:s0    0 Aug 24 16:30 .lock
-rw-r--r--.   1 root root system_u:object_r:unlabeled_t:s0   38 Aug 24 16:30 config
drwxr-sr-x.   2 root root system_u:object_r:unlabeled_t:s0    3 Aug 24 16:30 extensions
drwxr-sr-x. 258 root root system_u:object_r:unlabeled_t:s0 3547 Aug 24 16:30 objects
drwxr-sr-x.   5 root root system_u:object_r:unlabeled_t:s0   58 Aug 24 16:30 refs
drwxr-sr-x.   2 root root system_u:object_r:unlabeled_t:s0    3 Aug 24 16:30 state
drwxr-sr-x.   3 root root system_u:object_r:unlabeled_t:s0   28 Aug 24 16:30 tmp

[cgwalters]

Right, it's just the file descriptor being passed by the daemon. I think the more robust fix is to pass the FD directly over a private socket between client and daemon - we actually do make a private socket already for each transaction after it starts.

Possibly an even better fix is something like a StartTransaction() dbus method and then we support grouping multiple operations as part of a single real "transaction". That way we could even make what the MCD is doing today feel like one operation even if it's multiple rpm-ostree ... calls.