search

k3s-io / k3s-selinux

SELinux policy for k3s
Apache License 2.0
69 stars 20 forks source link

Incompatible with container-selinux > 2.188 #36

Open ameyp opened 1 year ago

ameyp commented 1 year ago

I'm running Fedora CoreOS 37.20221211.3.0, and I've installed both container-selinux-2.193.0 as well as k3s-selinux-1.2-2.el8 (from https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el8.noarch.rpm). However, the k3s module doesn't appear to be getting loaded on the system, the output of semodule -l does not contain k3s. Additionally, if I try to install the module manually with semodule -i <path>, it fails due to a conflict:

[root@localhost core]# semodule -i /usr/share/selinux/packages/k3s.pp
Conflicting name type transition rules
Binary policy creation failed at /etc/selinux/targeted/tmp/modules/400/k3s/cil:135
Failed to generate binary
semodule:  Failed!

The error leads me to believe that one of rules specified in this package is conflicting with a rule in one of the existing modules.

ameyp commented 1 year ago

The actual problem that I'm trying to solve, is that /var/lib/rancher/k3s/storage is being created with the label container_var_lib_t instead of container_file_t, thereby causing the local-path provisioner to fail with permission denied.

wranders commented 1 year ago

Having this problem as well. From the CIL, the offending statement is

(typetransition container_runtime_t container_var_lib_t dir "snapshots" container_share_t)

which comes from https://github.com/k3s-io/k3s-selinux/blob/d97be9f6fcf2c467f7e826bc2c8c58da1375bfde/policy/centos8/k3s.te#L36

I tried looking through container-selinux for anything that would conflict from the previous integration of k3s rules, but didn't find anything.

This is occurring in 37.20221225.3.0 as well. Here are the package versions used:

These were manually upgraded to the latest release version and made no difference.

mysticaltech commented 1 year ago

@brandond Having the exact same issue on OpenSUSE MicroOS latest!

Our project just came to an halt. See https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/528#issuecomment-1401700309

When trying with semodule -vv -i /usr/share/selinux/packages/k3s.pp

It gives those issues:

ksnip_20230125-004821

Please, how do we fix this? It's kind of urgent as many daily installs are now failing.

brandond commented 1 year ago

The EL8 policy should work with container-selinux 2.167.0-1 and selinux-policy 3.14.3-67, which seems to work fine on both EL8 and EL9. We don't technically support CoreOS so I can't speak as to specific compatibility with any of the packages on that distro.

For MicroOS, it should work with container-selinux 2.164.2-1.1 and selinux-policy 20210716-3.1. If you're having trouble with MicroOS, please verify what version of those packages you're using. I suspect that something has been updated on the MicroOS side that is now conflicting, but I don't track that distro closely enough to know what off the top of my head.

mysticaltech commented 1 year ago

@brandond Yes, we suspected that too, as it was working yesterday, and since the latest release that just dropped, the problem showed up!

Will have a look at those packages' versions, and at ways to downgrade them.

brandond commented 1 year ago

With regards to selinux and local-path-provisioner, that is maintained by another team - the same team that owns Longhorn, to be specific. I believe they have an issue in their backlog to improve its compatibility with selinux and PSA.

mysticaltech commented 1 year ago

@brandond Indeed the versions on MicroOS are significantly more recent. I will investigate which of these packages were updated since yesterday.

ksnip_20230125-011711

mysticaltech commented 1 year ago

@brandond Thanks for your tips, as always very helpful!

After investigating the aforementioned packages here https://download.opensuse.org/tumbleweed/repo/oss/noarch/. We can see that it was container-linux that was upgraded just recently from 2.188 to 2.198.

So, downgrading with transactional-update --no-selfupdate -n pkg install --oldpackage https://download.opensuse.org/tumbleweed/repo/oss/noarch/container-selinux-2.188.0-2.1.noarch.rpm did work! 🎉

After that, applying semodule -vv -i /usr/share/selinux/packages/k3s.pp does successfully build the policy binary.

ksnip_20230125-015211

However, I blocked future automated upgrades of container-selinux with --no-selfupdate to avoid issues upon automatic upgrade back to 2.198 when the node is running.

But will that affect the security of the node and/or future versions of k3s? And is it really necessary to do so (lock the version)? Just so you understand the context, our nodes on Kube-Hetzner are meant to remain evergreen, by combining MicroOS's automatic upgrades, and the system-upgrade-controller for k3s.

Since the policy has been built successfully already, if we leave the automatic upgrade on for that package, would it fail to run under the more recent container-selinux 2.198 ?

brandond commented 1 year ago

The updates are listed at https://build.opensuse.org/request/show/1059620 but I don't see anything obvious that would break it, so I'll take some additional research. I believe that some of the suspicions I saw about the problem being the same as whatever's affecting Fedora CoreOS are correct.

mysticaltech commented 1 year ago

Ok, good to know! If you need any command outputs or logs, please don't hesitate! 🙏

mysticaltech commented 1 year ago

Sadly, it is confirmed; even the automatic upgrade breaks it because it tries to rebuild the policy! So locking of the version is important until the definite fix is shipped.

ksnip_20230125-031952

galal-hussein commented 1 year ago

I can reproduce the issue with the following versions:

Compiling the policy revealed the same error, and the contexts are not applied to the processes or the files:

ls -ltrZ /var/lib/rancher/k3s/
total 4
drwxr-xr-x. 3 root root unconfined_u:object_r:var_lib_t:s0  106 Jan 25 19:42 data
drwxr-xr-x. 5 root root unconfined_u:object_r:var_lib_t:s0 4096 Jan 25 19:42 agent
drwx------. 8 root root system_u:object_r:var_lib_t:s0      148 Jan 25 19:42 server

Investigating the root cause now

galal-hussein commented 1 year ago

Please correct me if anything seems inaccurate or wrong, but I think I found the root cause, as @wranders mentioned, the problem was in that line:

Binary policy creation failed at /var/lib/selinux/targeted/tmp/modules/400/k3s/cil:135
...
(typetransition container_runtime_t container_var_lib_t dir "snapshots" container_share_t)

so looking further into what changed recently in container-selinux, I found this commit which seems to be the problem:

https://github.com/containers/container-selinux/commit/8ca4b89b82846d126000601f6b8530a962f0ecd3#diff-d87cfd6c86daf9204f212619e52210d06d7575d9b4ad27b41966ab3e1b80e2a8R536

specifically the conflict line will be here:

  filetrans_pattern($1, container_var_lib_t, container_file_t, dir, "snapshots")

which makes the snapshot dir writable, at first I was confused why this conflict didnt happen before in previous versions since the original line in container-selinux was:

 filetrans_pattern($1, container_var_lib_t, container_ro_file_t, dir, "snapshots")

and I found that container_share_t is basically an alias to container_ro_file_t:

typealias container_ro_file_t alias { container_share_t docker_share_t };

So I think the suggested fix will be changing the snapshots filetranspattern to match the container-selinux one.

cc @brandond @AkihiroSuda

brandond commented 1 year ago

Good find! I think we'll need to figure out how to handle this strategically, given that we do not need and cannot land this fix for EL8 or SLE Micro users (as those don't have the updated container-selinux version), but we don't have separate yum repos for EL8/Fedora Coreos and SLE Micro/MicroOS. We may need to create additional repos and add detection support to the install script.

mysticaltech commented 1 year ago

@brandond Not sure how it all works, but if the updated k3s-selinux RPM with the fix, could live on the rpm-testing.rancher.io repo for some time, it would be awesome!

AkihiroSuda commented 1 year ago

So I think the suggested fix will be changing the snapshots filetranspattern to match the container-selinux one.

Sorry what is the suggested fix? s/var_lib_t/container_file_t/ ? (Not container_ro_file_t, as the active snapshots have to be writable)

mysticaltech commented 1 year ago

FYI folks, the source PR https://github.com/containers/container-selinux/pull/82, where they explain their thinking.

brandond commented 1 year ago

if the updated k3s-selinux RPM with the fix, could live on the rpm-testing.rancher.io repo for some time, it would be awesome!

The fix would be appropriate to land there for MicroOS, but not for EL8.

mysticaltech commented 1 year ago

Just FYI @galal-hussein, I built it locally (from #37) for microos and installed on lastest MicroOS running container-selinux 2.198, and it worked!

However, I did notice this new relabeling activity going on, which was not there before to my knowledge. Please let me know if it's ok 🙏

ksnip_20230126-070408

And the final re-building of the policy (we do it after installing k3s, just in case):

ksnip_20230126-072441

galal-hussein commented 1 year ago

@AkihiroSuda yes the suggested fix will label s/var_lib_t/container_file_t/ instead of container_share_t https://github.com/k3s-io/k3s-selinux/pull/37/files#diff-a91aed3ea17f6544e61fe4e39d429a27316a95d80b7df73c3c43c76c9261a249R9

galal-hussein commented 1 year ago

Just FYI @galal-hussein, I built it locally (from #37) for microos and installed on lastest MicroOS running container-selinux 2.198, and it worked!

However, I did notice this new relabeling activity going on, which was not there before to my knowledge. Please let me know if it's ok pray

ksnip_20230126-070408

And the final re-building of the policy (we do it after installing k3s, just in case):

ksnip_20230126-072441

@mysticaltech good to know that its working, however I am not really sure about this relabeling activity, can you post the the final labels of these dirs for example:

mysticaltech commented 1 year ago

@galal-hussein Here you go:

ksnip_20230126-182608

galal-hussein commented 1 year ago

@mysticaltech the labels looks fine as far as I can see

mysticaltech commented 1 year ago

@galal-hussein Perfect, thanks for confirming!

Maybe this relabeling happens during the installation of the RPM because we actually install k3s after (because of the need to reboot into the new MicroOS snapshot). So the k3s directories are not there yet.

And after having rebooted into the new snapshot, we install k3s without SELinux, and immediately build the policy again with semodule as shown in the second screenshot up the thread.

mhjacks commented 1 year ago

It appears we also have the same issue on CentOS 9 Stream which (at this time) has container-selinux-2.198.0-1.el9.noarch.

mysticaltech commented 1 year ago

@galal-hussein Very happy to see the 1.3.testing, just so we know what to expect, is this coming to stable anytime soon?

rancher-max commented 1 year ago

This is being tested currently and we'll update this issue with results as we find them! Hoping to be able to have it stable by March or April patch releases, pending results.

johanneskastl commented 1 year ago

Any news here? rke2-selinux has a fix in the works...

balonik commented 1 year ago

This bug affects "normal" Fedora as well.

  Running scriptlet: container-selinux-2:2.211.1-1.fc37.noarch 
  Upgrading        : container-selinux-2:2.211.1-1.fc37.noarch       
  Running scriptlet: container-selinux-2:2.211.1-1.fc37.noarch    
Conflicting name type transition rules
Binary policy creation failed at /var/lib/selinux/targeted/tmp/modules/200/k3s/cil:135
Failed to generate binary
/usr/sbin/semodule:  Failed!
  Cleanup          : container-selinux-2:2.209.0-1.fc37.noarch      
  Running scriptlet: container-selinux-2:2.209.0-1.fc37.noarch
  Verifying        : container-selinux-2:2.211.1-1.fc37.noarch      
  Verifying        : container-selinux-2:2.209.0-1.fc37.noarch
eriklogtenberg commented 1 year ago

Same on Fedora 38. These are the two packages the k3s installer wants to install:

 Uitvoeren van scriptlet: k3s-selinux-1.2-2.el8.noarch                                                                                                    2/2 
 Installeren            : k3s-selinux-1.2-2.el8.noarch                                                                                                    2/2 
 Uitvoeren van scriptlet: k3s-selinux-1.2-2.el8.noarch                                                                                                    2/2 
Conflicting name type transition rules
Binary policy creation failed at /var/lib/selinux/targeted/tmp/modules/200/k3s/cil:135
Failed to generate binary
/usr/sbin/semodule:  Failed!
marnickv commented 1 year ago

Same here on a fresh Fedora 38 server.

Running transaction
  Preparing        :                                                                                                                                                                                                                                          1/1 
  Running scriptlet: container-selinux-2:2.211.1-1.fc38.noarch                                                                                                                                                                                                1/2 
  Installing       : container-selinux-2:2.211.1-1.fc38.noarch                                                                                                                                                                                                1/2 
  Running scriptlet: container-selinux-2:2.211.1-1.fc38.noarch                                                                                                                                                                                                1/2 
  Running scriptlet: k3s-selinux-1.2-2.el8.noarch                                                                                                                                                                                                             2/2 
  Installing       : k3s-selinux-1.2-2.el8.noarch                                                                                                                                                                                                             2/2 
  Running scriptlet: k3s-selinux-1.2-2.el8.noarch                                                                                                                                                                                                             2/2 
Conflicting name type transition rules
Binary policy creation failed at /var/lib/selinux/targeted/tmp/modules/200/k3s/cil:135
Failed to generate binary
/usr/sbin/semodule:  Failed!
brandond commented 1 year ago

The updated k3s-selinux package has not yet been released to the stable channel, pending QA validation. If you'd like to try it out, please install the package from https://github.com/k3s-io/k3s-selinux/releases/tag/v1.3.testing.7 or by using INSTALL_K3S_CHANNEL=testing when using the k3s install script.

philipsparrow commented 1 year ago

Thanks @brandond. I just tried installing 1.3-7.slemicro onto MicroOS (with container-selinux 2.211.0-1.1 and selinux-policy-20230425-1.1) and it complains that nothing provides 'container-selinux < 2.191.0-1'. See slemicro/k3s-selinux.spec

I installed anyway, it still failed building the binary policy and still didn't work after rebooting and re-running semodule on k3s.pp

johanneskastl commented 1 year ago

Thanks @brandond. I just tried installing 1.3-7.slemicro onto MicroOS (with container-selinux 2.211.0-1.1 and selinux-policy-20230425-1.1) and it complains that nothing provides 'container-selinux < 2.191.0-1'. See slemicro/k3s-selinux.spec

@philipsparrow Can you try the RPM from https://download.opensuse.org/repositories/home:/ojkastl_buildservice:/Branch_devel_microos/openSUSE_Tumbleweed/? This is the package for openSUSE, based on the 1.3.testing.7 tag.

philipsparrow commented 1 year ago

@johanneskastl that package installed without issue and I re-ran semodule -v -i /usr/share/selinux/packages/k3s.pp to confirm that there was no failure.

I then installed k3s and it was mostly working.

Unfortunately my original issue still persists: I'm still seeing an issue with local-path-provisioner being unable to create /var/lib/rancher/k3s/storage (to create PVCs) because the directory has system_u:object_r:var_lib_t:s0 rather than system_u:object_r:container_file_t:s0. Perhaps this is a separate issue; I'm going to rebuild my VM and try afresh with your RPM.

philipsparrow commented 1 year ago

@johanneskastl it's working with your RPM. Will this RPM be making its way into the stable channel?

mysticaltech commented 1 year ago

@philipsparrow This one should work too https://github.com/k3s-io/k3s-selinux/releases/download/v1.3.testing.7/k3s-selinux-1.3-7.sle.noarch.rpm, and yes as @brandond announced, it's on its way to stable soon, pending some QA.

rancher-max commented 1 year ago

@philipsparrow regarding this comment:

Unfortunately my original issue still persists: I'm still seeing an issue with local-path-provisioner being unable to create /var/lib/rancher/k3s/storage (to create PVCs) because the directory has system_u:object_r:var_lib_t:s0 rather than system_u:object_r:container_file_t:s0.

Did you reboot your system after install? It's easy to miss, but this should be in the output of the install script now:

$ curl -sfL https://raw.githubusercontent.com/galal-hussein/k3s/add_el9_selinux_rpm/install.sh | INSTALL_K3S_CHANNEL=testing sh -
...
Added:
  k3s-selinux-1.3-7.coreos.noarch
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[WARN]  Please reboot your machine to activate the changes and avoid data loss.
...

Without this reboot, most of the selinux contexts don't write appropriately. However with it, it seems to be working for me in my tests:

$ sudo ls -laZ /var/lib/rancher/k3s/storage/
total 0
drwx-----x. 3 root root system_u:object_r:container_file_t:s0    77 May 16 23:24 .
drwxr-xr-x. 6 root root system_u:object_r:container_var_lib_t:s0 60 May 16 23:24 ..
drwxrwxrwx. 2 root root system_u:object_r:container_file_t:s0     6 May 16 23:24 pvc-34b4fcf7-7d5a-4e37-9fb1-062749b8dbeb_default_local-path-pvc

The conditions of the OS for this test are as follows, but the same should be true for container-selinux 2.211:

$ uname -a && echo "" && cat /etc/os-release 
Linux ip-172-31-34-115 6.2.13-300.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 27 01:33:30 UTC 2023 x86_64 GNU/Linux

NAME="Fedora Linux"
VERSION="38.20230430.1.0 (CoreOS)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora CoreOS 38.20230430.1.0"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:38"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='38.20230430.1.0'
$ rpm -qi container-selinux
Name        : container-selinux
Epoch       : 2
Version     : 2.209.0
Release     : 1.fc38
Architecture: noarch
Install Date: Tue May  2 16:28:16 2023
Group       : Unspecified
Size        : 64036
License     : GPL-2.0-only
Signature   : RSA/SHA256, Mon Apr  3 08:32:18 2023, Key ID 809a8d7ceb10b464
Source RPM  : container-selinux-2.209.0-1.fc38.src.rpm
Build Date  : Mon Apr  3 07:21:36 2023
Build Host  : buildvm-s390x-25.s390.fedoraproject.org
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://github.com/containers/container-selinux
Bug URL     : https://bugz.fedoraproject.org/container-selinux
Summary     : SELinux policies for container runtimes
Description :
SELinux policy modules for use with container runtimes.

Hoping to have all validations for this completed by tonight or tomorrow (so far the results have been promising), and then we will push out the changes to latest channels, followed a few days later by pushing to stable channels assuming you all in the community don't see issues using latest 😄

philipsparrow commented 1 year ago

This is wonderful news!

My challenge is to ensure it works inside a Combustion first-boot script with a custom mirror. What I do is install the dependencies (e.g. container-selinux, k3s-selinux) first and then run k3s-install (with INSTALL_K3S_SKIP_SELINUX_RPM=true). I'll try an end-to-end test and try the manual reboot.

philipsparrow commented 1 year ago

This worked for me without reboot, as long as I installed k3s (inside Combustion/Ignition) with:

INSTALL_K3S_SELINUX_WARN=true
INSTALL_K3S_SKIP_SELINUX_RPM=true

Probably worked without a reboot because after Combustion (Ignition first-boot) completes, it loads the transactional snapshot. Only errors I see are about invalid ELF headers and unlikely to be related to k3s-selinux. ls -Z looks good, PVs get created, k3s binary upgrade works via Rancher.

Thanks very much for the help.

jobcespedes commented 1 year ago

Hi,

I had to run restorecon -Rv /var/lib/rancher/k3s/storage for correct labels to be applied for local path storage. I am testing using clean installs. First, latest k3s-selinux is installed. Then k3s is installed. No luck so far. Then I run restorecon and it works.

Some info:

NAME="Fedora Linux"
VERSION="38.20230414.3.0 (CoreOS)"

k3s-selinux-1.3-8.coreos.noarch
container-selinux-2.209.0-1.fc38.noarch
rancher-max commented 1 year ago

Hi everyone! I wrote some more details on what was tested in https://github.com/k3s-io/k3s/issues/7307#issuecomment-1562078486, but the tl;dr is that this should be working and available now in the testing channel with the default install script. I will leave this open until we make that available in latest and stable, which will be in line with the May k3s releases (very soon).

Klaas- commented 1 year ago

Feedback for RHEL8: k3s-selinux-1.3-1.el8.noarch works with container-selinux-2:2.205.0-2.module+el8.8.0+18438+15d3aa65.noarch, but I had to delete k3s-selinux, upgrading it was not enough.

mysticaltech commented 1 year ago

@galal-hussein So the latest stable fixes this completely, correct?