Open cyrus-mc opened 8 years ago
cc @mjg59 @steveeJ @jonboulle
From irc discussion, @cyrus-mc is using stage1-coreos.aci
hmm, is this the v0.10.0 release from GitHub, or did you build it yourself? Could you please paste the entire output of what happens when you try to run it?
I've gotten the same issue when using rkt v0.10.0 and v0.11.0 (from releases page) on Fedora 23. I have to disable SELinux to run pods.
Run output:
$ sudo rkt run --insecure-skip-verify sha512-3f25067fc69f
rkt: using image from local store for image name coreos.com/rkt/stage1-coreos:0.11.0
/usr/lib/systemd/systemd: error while loading shared libraries: libselinux.so.1: cannot open shared object file: Permission denied
Full audit logs here.
Interesting part:
type=AVC msg=audit(1447866807.475:2032): avc: denied { read } for pid=11851 comm="systemd" name="libselinux.so.1" dev="dm-2" ino=6569480 scontext=system_u:system_r:svirt_lxc_net_t:s0:c330,c605 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
type=AVC msg=audit(1447866807.475:2033): avc: denied { read } for pid=11851 comm="systemd" name="libselinux.so.1" dev="dm-2" ino=6569480 scontext=system_u:system_r:svirt_lxc_net_t:s0:c330,c605 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
type=USER_AVC msg=audit(1447866807.516:2034): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='Unknown permission stop for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
type=USER_AVC msg=audit(1447866807.517:2035): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='Unknown permission stop for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
@mjg59 any ideas here?
From OOB communication with @mjg59 we can say that rkt
is not supported by Fedora/RHEL/CentOS selinux policy.
So, either we add a disable-selinux option to configure or we wait until Fedora policy gets fixed.
Another issue might be that we expect overlays to support selinux and CoreOS has a patched kernel for that.
The overlay fs patch for SELinux support is here with others in the series.
@lsm5 @rhatdan @mjg59
The command ausearch -m avc -ts recent
returns:
----
time->Tue Jan 19 14:54:36 2016
type=AVC msg=audit(1453211676.627:617): avc: denied { read } for pid=7201 comm="systemd" name="libselinux.so.1" dev="dm-1" ino=541346 scontext=system_u:system_r:svirt_lxc_net_t:s0:c9,c955 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
----
time->Tue Jan 19 14:54:36 2016
type=AVC msg=audit(1453211676.627:618): avc: denied { read } for pid=7201 comm="systemd" name="libselinux.so.1" dev="dm-1" ino=541346 scontext=system_u:system_r:svirt_lxc_net_t:s0:c9,c955 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
So it is systemd inside the container trying to read libselinux.so.1 inside the container. ls -liZ
returns:
541346 -rwxr-xr-x. 1 root root unconfined_u:object_r:var_lib_t:s0 145440 Sep 9 07:44 /var/lib/rkt/cas/tree/deps-sha512-f5143c3cd24d997aea9a1c0e4e641f4ed9a44edeb4fdfe231a3b8059475100ca/rootfs/usr/lib64/libselinux.so.1
541346 -rwxr-xr-x. 1 root root system_u:object_r:svirt_sandbox_file_t:s0:c9,c955 145440 Sep 9 07:44 /var/lib/rkt/pods/run/0570eb27-fba5-4a2d-8550-9becf0d9ec70/stage1/rootfs/usr/lib64/libselinux.so.1
They are using the same inode because the first one is the overlay fs lower directory, and the second one is in the overlay fs mount.
Same error without using overlay fs:
type=AVC msg=audit(1453212466.826:869): avc: denied { read } for pid=9315 comm="systemd" name="libselinux.so.1" dev="dm-1" ino=541825 scontext=system_u:system_r:svirt_lxc_net_t:s0:c581,c949 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
541825 -rwxr-xr-x. 1 root root unconfined_u:object_r:var_lib_t:s0 145440 Sep 9 07:44 /var/lib/rkt/pods/run/2787d853-c031-483f-a400-450a7be183d0/stage1/rootfs/usr/lib64/libselinux.so.1
You can not use SELinux and Overlay at this time. Well I guess you could label everything under /var/lib/rkt/pods as system_u:object_r:svirt_sandbox_file_t:s0 and it might work better. Not sure how you disable SELinux for rkt process separation, but you might have to. Red Hat Kernel Engineers continue to look to improve OverlayFS with SELinux but nothing so far.
@rhatdan I tested as well with rkt run --no-overlay
so that rkt does not use overlay fs, but I get the same error, see comment above.
Then, I tried:
$ sudo chcon -R -u system_u -t svirt_sandbox_file_t /var/lib/rkt/pods
$ sudo ./build-rkt-0.15.0+git/bin/rkt run --debug --insecure-options=image --interactive --no-overlay docker://busybox
(...)
Welcome to Linux!
Initializing machine ID from container UUID.
bind(/run/systemd/notify) failed: Permission denied
Failed to bind private socket: Permission denied
Failed to fully start up daemon: No such file or directory
Failed to add watch on /run/systemd/ask-password: No such file or directory
So it works a bit further but still fails. ausearch -m avc -ts recent
has plenty of errors; the first ones are:
----
time->Wed Jan 20 16:03:38 2016
type=AVC msg=audit(1453302218.553:2106): avc: denied { write } for pid=4329 comm="systemd" name="systemd" dev="tmpfs" ino=682171 scontext=system_u:system_r:svirt_lxc_net_
t:s0:c433,c492 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0
----
time->Wed Jan 20 16:03:38 2016
type=AVC msg=audit(1453302218.553:2107): avc: denied { write } for pid=4329 comm="systemd" name="systemd" dev="tmpfs" ino=682171 scontext=system_u:system_r:svirt_lxc_net_
t:s0:c433,c492 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0
----
time->Wed Jan 20 16:03:38 2016
type=AVC msg=audit(1453302218.553:2108): avc: denied { read write } for pid=4329 comm="systemd" name="tty" dev="tmpfs" ino=682156 scontext=system_u:system_r:svirt_lxc_net
_t:s0:c433,c492 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=chr_file permissive=0
----
This looks like this content "systemd dir" and "tty chr_file" are being created by the user process on /tmp and then somehow being used within the container.
I guess rkt guys will need to figure out what is happening, I have not used rkt.
Please update the "Getting Started with rkt Containers in 3 minutes post" to mention this issue for Fedora users
https://coreos.com/blog/getting-started-with-rkt-1.0.html
I'm both super excited to try rkt and WAY past 3 minutes debugging this SElinux issue on Fedora 23. @alban summarizes my state above exactly, so there is nothing to add technically, but as a user I'm frustrated having gotten to step 2 and had to start debugging.
Especially noting that the post says "rkt is packaged for a few Linux distributions, like ... Fedora Rawhide", made me think that even if I'm not on rawhide, I can just wget the sources. There is no mention of needing additional SELinux policies installed.
It seems the last successful build of Fedora-Cloud-Base-rawhide was done on 20160222, so it does not contain the fixes in fedora-selinux/selinux-policy#100.
This version (Fedora-Cloud-Base-rawhide-20160222.x86_64) has selinux-policy-targeted and selinux-policy 3.13.1-171.fc24 but I can update to 3.13.1-175.fc25. Is it enough to get the fix?
I tried with rkt git master (including #2247) and selinux-policy-3.13.1-175.fc25.noarch and selinux-policy-targeted-3.13.1-175.fc25.noarch but I still have the error:
Press ^] three times within 1s to kill container.
Container rkt-14d03f8b-fc70-4567-94d8-630baa3e39e6 failed with error code 127.
/usr/lib/systemd/systemd: error while loading shared libraries: libselinux.so.1: cannot open shared object file: Permission denied
It's systemd in the container:
-rwxr-xr-x. 1 root root unconfined_u:object_r:var_lib_t:s0 1507304 Sep 9 06:44 /var/lib/rkt/pods/run/de64f4cc-338e-49a5-a6fd-46ffc8ec76e7/stage1/rootfs/usr/lib64/systemd/systemd
-rwxr-xr-x. 1 root root unconfined_u:object_r:var_lib_t:s0 145440 Sep 9 05:44 /var/lib/rkt/pods/run/de64f4cc-338e-49a5-a6fd-46ffc8ec76e7/stage1/rootfs/usr/lib64/libselinux.so.1
[pid 1595] execve("/usr/lib/systemd/systemd", ["/usr/lib/systemd/systemd", "--default-standard-output=tty"], [/* 7 vars */]) = 0
...
[pid 1595] open("/usr/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
type=AVC msg=audit(1457356790.646:410): avc: denied { read } for pid=1435
comm="systemd" name="libselinux.so.1" dev="xvda1" ino=151969
scontext=system_u:system_r:svirt_lxc_net_t:s0:c305,c666
tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
This looks like you don't have the rkt patch that relabels the content under /var/lib/rkt/pods/run correctly.
Status: with overlay, we still have this error on Fedora. Without overlay, this error is gone.
Yes, we are working daily on the Kernel to fix the OverlayFS v SELinux issues. Currently doing it privately (With Stephen Smalley NSA), since do not want to deal with the noise until we have locked down a proposal.
@rhatdan do you have a status update on this?
It is still ongoing. This is a fairly complex issue, and getting the security right is critical important. Met with the kernel engineer this morning and we have a working plan on how to handle access checking on CopyUp as well as when a context=LABEL option is used. Once he has a working demo, he will put the patches up for review.
Removing from specific milestone as still dependent on external work.
am having this same issue in Fedora 23
am having this same issue in Fedora 23
The fixes (https://github.com/fedora-selinux/selinux-policy/pull/100 and https://github.com/fedora-selinux/selinux-policy/pull/108) are in Fedora 24. There's one more fix (https://github.com/fedora-selinux/selinux-policy/pull/114) which I'm not sure it's included in 24, but it is in Rawhide.
The pending kernel patches for SELinux + Overlayfs: [RFC PATCH 0/9][V3] Overlayfs SELinux Support
/cc @mjg59
Yes, it currently looks like this will land in 4.9 kernel.
is this resolved?
SELinux and Overlayfs are currently working in the Fedora Rawhide Kernel and Fedora 25.
Overlayfs supports selinux since Linux 4.9.
The related commits are the following:
a518b0a selinux: Implement dentry_create_files_as() hook
2602625 security, overlayfs: Provide hook to correctly label newly created
files
c957f6d selinux: Pass security pointer to determine_inode_label()
19472b6 selinux: Implementation for inode_copy_up_xattr() hook
121ab82 security,overlayfs: Provide security hook for copy up of xattrs for
overlay file
56909eb selinux: Implementation for inode_copy_up() hook
d8ad8b4 security, overlayfs: provide copy up security hook for unioned files
Environment
OS: CentOS 7.1 Kernel: 3.10.0 rkt: 0.10.0 systemd: 208
When trying to start any container, with selinux set to Enforcing I received the following error:
/usr/lib/systemd/systemd: error while loading shared libraries: libselinux.so.1: cannot open shared object file: Permission denied
Disabling selinux (disabled or Permissive) solved the issue.