search

rkt / rkt

[Project ended] rkt is a pod-native container engine for Linux. It is composable, secure, and built on standards.
Apache License 2.0
8.82k stars 886 forks source link

SElinux not correctly working with overlayfs #1727

Open cyrus-mc opened 8 years ago

cyrus-mc commented 8 years ago

Environment

OS: CentOS 7.1 Kernel: 3.10.0 rkt: 0.10.0 systemd: 208

When trying to start any container, with selinux set to Enforcing I received the following error:

/usr/lib/systemd/systemd: error while loading shared libraries: libselinux.so.1: cannot open shared object file: Permission denied

Disabling selinux (disabled or Permissive) solved the issue.

yifan-gu commented 8 years ago

cc @mjg59 @steveeJ @jonboulle

yifan-gu commented 8 years ago

From irc discussion, @cyrus-mc is using stage1-coreos.aci

jonboulle commented 8 years ago

hmm, is this the v0.10.0 release from GitHub, or did you build it yourself? Could you please paste the entire output of what happens when you try to run it?

ericchiang commented 8 years ago

I've gotten the same issue when using rkt v0.10.0 and v0.11.0 (from releases page) on Fedora 23. I have to disable SELinux to run pods.

Run output:

$ sudo rkt run --insecure-skip-verify sha512-3f25067fc69f
rkt: using image from local store for image name coreos.com/rkt/stage1-coreos:0.11.0
/usr/lib/systemd/systemd: error while loading shared libraries: libselinux.so.1: cannot open shared object file: Permission denied

Full audit logs here.

Interesting part:

type=AVC msg=audit(1447866807.475:2032): avc:  denied  { read } for  pid=11851 comm="systemd" name="libselinux.so.1" dev="dm-2" ino=6569480 scontext=system_u:system_r:svirt_lxc_net_t:s0:c330,c605 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
type=AVC msg=audit(1447866807.475:2033): avc:  denied  { read } for  pid=11851 comm="systemd" name="libselinux.so.1" dev="dm-2" ino=6569480 scontext=system_u:system_r:svirt_lxc_net_t:s0:c330,c605 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
type=USER_AVC msg=audit(1447866807.516:2034): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='Unknown permission stop for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
type=USER_AVC msg=audit(1447866807.517:2035): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='Unknown permission stop for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
jonboulle commented 8 years ago

@mjg59 any ideas here?

krnowak commented 8 years ago

From OOB communication with @mjg59 we can say that rkt is not supported by Fedora/RHEL/CentOS selinux policy.

So, either we add a disable-selinux option to configure or we wait until Fedora policy gets fixed.

Another issue might be that we expect overlays to support selinux and CoreOS has a patched kernel for that.

alban commented 8 years ago

The overlay fs patch for SELinux support is here with others in the series.

alban commented 8 years ago

@lsm5 @rhatdan @mjg59

The command ausearch -m avc -ts recent returns:

----
time->Tue Jan 19 14:54:36 2016
type=AVC msg=audit(1453211676.627:617): avc:  denied  { read } for  pid=7201 comm="systemd" name="libselinux.so.1" dev="dm-1" ino=541346 scontext=system_u:system_r:svirt_lxc_net_t:s0:c9,c955 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
----
time->Tue Jan 19 14:54:36 2016
type=AVC msg=audit(1453211676.627:618): avc:  denied  { read } for  pid=7201 comm="systemd" name="libselinux.so.1" dev="dm-1" ino=541346 scontext=system_u:system_r:svirt_lxc_net_t:s0:c9,c955 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0

So it is systemd inside the container trying to read libselinux.so.1 inside the container. ls -liZ returns:

541346 -rwxr-xr-x. 1 root root unconfined_u:object_r:var_lib_t:s0                145440 Sep  9 07:44 /var/lib/rkt/cas/tree/deps-sha512-f5143c3cd24d997aea9a1c0e4e641f4ed9a44edeb4fdfe231a3b8059475100ca/rootfs/usr/lib64/libselinux.so.1
541346 -rwxr-xr-x. 1 root root system_u:object_r:svirt_sandbox_file_t:s0:c9,c955 145440 Sep  9 07:44 /var/lib/rkt/pods/run/0570eb27-fba5-4a2d-8550-9becf0d9ec70/stage1/rootfs/usr/lib64/libselinux.so.1

They are using the same inode because the first one is the overlay fs lower directory, and the second one is in the overlay fs mount.

alban commented 8 years ago

Same error without using overlay fs:

type=AVC msg=audit(1453212466.826:869): avc:  denied  { read } for  pid=9315 comm="systemd" name="libselinux.so.1" dev="dm-1" ino=541825 scontext=system_u:system_r:svirt_lxc_net_t:s0:c581,c949 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0

541825 -rwxr-xr-x. 1 root root unconfined_u:object_r:var_lib_t:s0 145440 Sep  9 07:44 /var/lib/rkt/pods/run/2787d853-c031-483f-a400-450a7be183d0/stage1/rootfs/usr/lib64/libselinux.so.1
rhatdan commented 8 years ago

You can not use SELinux and Overlay at this time. Well I guess you could label everything under /var/lib/rkt/pods as system_u:object_r:svirt_sandbox_file_t:s0 and it might work better. Not sure how you disable SELinux for rkt process separation, but you might have to. Red Hat Kernel Engineers continue to look to improve OverlayFS with SELinux but nothing so far.

alban commented 8 years ago

@rhatdan I tested as well with rkt run --no-overlay so that rkt does not use overlay fs, but I get the same error, see comment above.

Then, I tried:

$ sudo chcon -R  -u system_u -t svirt_sandbox_file_t /var/lib/rkt/pods
$ sudo ./build-rkt-0.15.0+git/bin/rkt run --debug --insecure-options=image --interactive --no-overlay docker://busybox
(...)
Welcome to Linux!

Initializing machine ID from container UUID.
bind(/run/systemd/notify) failed: Permission denied
Failed to bind private socket: Permission denied
Failed to fully start up daemon: No such file or directory
Failed to add watch on /run/systemd/ask-password: No such file or directory

So it works a bit further but still fails. ausearch -m avc -ts recent has plenty of errors; the first ones are:

----
time->Wed Jan 20 16:03:38 2016
type=AVC msg=audit(1453302218.553:2106): avc:  denied  { write } for  pid=4329 comm="systemd" name="systemd" dev="tmpfs" ino=682171 scontext=system_u:system_r:svirt_lxc_net_
t:s0:c433,c492 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0
----
time->Wed Jan 20 16:03:38 2016
type=AVC msg=audit(1453302218.553:2107): avc:  denied  { write } for  pid=4329 comm="systemd" name="systemd" dev="tmpfs" ino=682171 scontext=system_u:system_r:svirt_lxc_net_
t:s0:c433,c492 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0
----
time->Wed Jan 20 16:03:38 2016
type=AVC msg=audit(1453302218.553:2108): avc:  denied  { read write } for  pid=4329 comm="systemd" name="tty" dev="tmpfs" ino=682156 scontext=system_u:system_r:svirt_lxc_net
_t:s0:c433,c492 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=chr_file permissive=0
----
rhatdan commented 8 years ago

This looks like this content "systemd dir" and "tty chr_file" are being created by the user process on /tmp and then somehow being used within the container.

I guess rkt guys will need to figure out what is happening, I have not used rkt.

erickj commented 8 years ago

Please update the "Getting Started with rkt Containers in 3 minutes post" to mention this issue for Fedora users

https://coreos.com/blog/getting-started-with-rkt-1.0.html

I'm both super excited to try rkt and WAY past 3 minutes debugging this SElinux issue on Fedora 23. @alban summarizes my state above exactly, so there is nothing to add technically, but as a user I'm frustrated having gotten to step 2 and had to start debugging.

Especially noting that the post says "rkt is packaged for a few Linux distributions, like ... Fedora Rawhide", made me think that even if I'm not on rawhide, I can just wget the sources. There is no mention of needing additional SELinux policies installed.

alban commented 8 years ago

https://github.com/fedora-selinux/selinux-policy/pull/100

alban commented 8 years ago

It seems the last successful build of Fedora-Cloud-Base-rawhide was done on 20160222, so it does not contain the fixes in fedora-selinux/selinux-policy#100.

This version (Fedora-Cloud-Base-rawhide-20160222.x86_64) has selinux-policy-targeted and selinux-policy 3.13.1-171.fc24 but I can update to 3.13.1-175.fc25. Is it enough to get the fix?

alban commented 8 years ago

I tried with rkt git master (including #2247) and selinux-policy-3.13.1-175.fc25.noarch and selinux-policy-targeted-3.13.1-175.fc25.noarch but I still have the error:

Press ^] three times within 1s to kill container.

Container rkt-14d03f8b-fc70-4567-94d8-630baa3e39e6 failed with error code 127.
/usr/lib/systemd/systemd: error while loading shared libraries: libselinux.so.1: cannot open shared object file: Permission denied

It's systemd in the container:

-rwxr-xr-x. 1 root root unconfined_u:object_r:var_lib_t:s0 1507304 Sep  9 06:44 /var/lib/rkt/pods/run/de64f4cc-338e-49a5-a6fd-46ffc8ec76e7/stage1/rootfs/usr/lib64/systemd/systemd
-rwxr-xr-x. 1 root root unconfined_u:object_r:var_lib_t:s0 145440 Sep  9 05:44 /var/lib/rkt/pods/run/de64f4cc-338e-49a5-a6fd-46ffc8ec76e7/stage1/rootfs/usr/lib64/libselinux.so.1
[pid  1595] execve("/usr/lib/systemd/systemd", ["/usr/lib/systemd/systemd", "--default-standard-output=tty"], [/* 7 vars */]) = 0
...
[pid  1595] open("/usr/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
type=AVC msg=audit(1457356790.646:410): avc:  denied  { read } for  pid=1435
comm="systemd" name="libselinux.so.1" dev="xvda1" ino=151969
scontext=system_u:system_r:svirt_lxc_net_t:s0:c305,c666
tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
rhatdan commented 8 years ago

This looks like you don't have the rkt patch that relabels the content under /var/lib/rkt/pods/run correctly.

alban commented 8 years ago

Status: with overlay, we still have this error on Fedora. Without overlay, this error is gone.

rhatdan commented 8 years ago

Yes, we are working daily on the Kernel to fix the OverlayFS v SELinux issues. Currently doing it privately (With Stephen Smalley NSA), since do not want to deal with the noise until we have locked down a proposal.

alban commented 8 years ago

@rhatdan do you have a status update on this?

rhatdan commented 8 years ago

It is still ongoing. This is a fairly complex issue, and getting the security right is critical important. Met with the kernel engineer this morning and we have a working plan on how to handle access checking on CopyUp as well as when a context=LABEL option is used. Once he has a working demo, he will put the patches up for review.

jonboulle commented 8 years ago

Removing from specific milestone as still dependent on external work.

sym3tri commented 8 years ago

am having this same issue in Fedora 23

iaguis commented 8 years ago

am having this same issue in Fedora 23

The fixes (https://github.com/fedora-selinux/selinux-policy/pull/100 and https://github.com/fedora-selinux/selinux-policy/pull/108) are in Fedora 24. There's one more fix (https://github.com/fedora-selinux/selinux-policy/pull/114) which I'm not sure it's included in 24, but it is in Rawhide.

alban commented 8 years ago

The pending kernel patches for SELinux + Overlayfs: [RFC PATCH 0/9][V3] Overlayfs SELinux Support

/cc @mjg59

rhatdan commented 8 years ago

Yes, it currently looks like this will land in 4.9 kernel.

jonboulle commented 7 years ago

is this resolved?

rhatdan commented 7 years ago

SELinux and Overlayfs are currently working in the Fedora Rawhide Kernel and Fedora 25.

wenjianhn commented 7 years ago

Overlayfs supports selinux since Linux 4.9.

The related commits are the following:

a518b0a selinux: Implement dentry_create_files_as() hook
2602625 security, overlayfs: Provide hook to correctly label newly created
files
c957f6d selinux: Pass security pointer to determine_inode_label()
19472b6 selinux: Implementation for inode_copy_up_xattr() hook
121ab82 security,overlayfs: Provide security hook for copy up of xattrs for
overlay file
56909eb selinux: Implementation for inode_copy_up() hook
d8ad8b4 security, overlayfs: provide copy up security hook for unioned files