search

rancher / rke2-selinux

RKE2 selinux + RPM packaging for selinux
Apache License 2.0
21 stars 21 forks source link

rke2-selinux 0.15-1 is broken on Rocky 8 #58

Open sfackler opened 1 year ago

sfackler commented 1 year ago 
# dnf install rke2-server
Rancher RKE2 Common Latest                                                                                                                                                                            2.6 kB/s | 2.6 kB     00:00
Rancher RKE2 1.28 Latest                                                                                                                                                                              2.6 kB/s | 1.9 kB     00:00
Dependencies resolved.
======================================================================================================================================================================================================================================
 Package                                             Architecture                                  Version                                                    Repository                                                         Size
======================================================================================================================================================================================================================================
Installing:
 rke2-server                                         x86_64                                        1.28.2~rke2r1-0.el8                                        rancher-rke2-1-28-latest                                          8.8 k
Installing dependencies:
 rke2-common                                         x86_64                                        1.28.2~rke2r1-0.el8                                        rancher-rke2-1-28-latest                                           20 M
 rke2-selinux                                        noarch                                        0.15-1.el8                                                 rancher-rke2-common-latest                                         21 k

Transaction Summary
======================================================================================================================================================================================================================================
Install  3 Packages

Total download size: 20 M
Installed size: 85 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): rke2-server-1.28.2~rke2r1-0.el8.x86_64.rpm                                                                                                                                                      60 kB/s | 8.8 kB     00:00
(2/3): rke2-selinux-0.15-1.el8.noarch.rpm                                                                                                                                                             124 kB/s |  21 kB     00:00
(3/3): rke2-common-1.28.2~rke2r1-0.el8.x86_64.rpm                                                                                                                                                     7.6 MB/s |  20 MB     00:02
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                 7.6 MB/s |  20 MB     00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                              1/1
  Running scriptlet: rke2-selinux-0.15-1.el8.noarch                                                                                                                                                                               1/3
  Installing       : rke2-selinux-0.15-1.el8.noarch                                                                                                                                                                               1/3
  Running scriptlet: rke2-selinux-0.15-1.el8.noarch                                                                                                                                                                               1/3
Failed to resolve typeattributeset statement at /var/lib/selinux/targeted/tmp/modules/400/rke2/cil:63
semodule:  Failed!

  Installing       : rke2-common-1.28.2~rke2r1-0.el8.x86_64                                                                                                                                                                       2/3
  Installing       : rke2-server-1.28.2~rke2r1-0.el8.x86_64                                                                                                                                                                       3/3
  Running scriptlet: rke2-server-1.28.2~rke2r1-0.el8.x86_64                                                                                                                                                                       3/3
  Running scriptlet: rke2-selinux-0.15-1.el8.noarch                                                                                                                                                                               3/3
  Running scriptlet: rke2-server-1.28.2~rke2r1-0.el8.x86_64                                                                                                                                                                       3/3
  Verifying        : rke2-selinux-0.15-1.el8.noarch                                                                                                                                                                               1/3
  Verifying        : rke2-common-1.28.2~rke2r1-0.el8.x86_64                                                                                                                                                                       2/3
  Verifying        : rke2-server-1.28.2~rke2r1-0.el8.x86_64                                                                                                                                                                       3/3

Installed:
  rke2-common-1.28.2~rke2r1-0.el8.x86_64                                         rke2-selinux-0.15-1.el8.noarch                                         rke2-server-1.28.2~rke2r1-0.el8.x86_64

Complete!
# ls -lZ /usr/bin/rke2
-rwxr-xr-x. 1 root root system_u:object_r:bin_t:s0 89186504 Sep 18 18:18 /usr/bin/rke2

Manually downgrading to 0.14 works:


# dnf install rke2-server rke2-selinux-0.14
Last metadata expiration check: 0:00:42 ago on Wed 11 Oct 2023 10:19:39 AM EDT.
Dependencies resolved.
======================================================================================================================================================================================================================================
 Package                                             Architecture                                  Version                                                    Repository                                                         Size
======================================================================================================================================================================================================================================
Installing:
 rke2-selinux                                        noarch                                        0.14-1.el8                                                 rancher-rke2-common-latest                                         21 k
 rke2-server                                         x86_64                                        1.28.2~rke2r1-0.el8                                        rancher-rke2-1-28-latest                                          8.8 k
Installing dependencies:
 rke2-common                                         x86_64                                        1.28.2~rke2r1-0.el8                                        rancher-rke2-1-28-latest                                           20 M

Transaction Summary
======================================================================================================================================================================================================================================
Install  3 Packages

Total download size: 20 M
Installed size: 85 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): rke2-server-1.28.2~rke2r1-0.el8.x86_64.rpm                                                                                                                                                      93 kB/s | 8.8 kB     00:00
(2/3): rke2-selinux-0.14-1.el8.noarch.rpm                                                                                                                                                              92 kB/s |  21 kB     00:00
(3/3): rke2-common-1.28.2~rke2r1-0.el8.x86_64.rpm                                                                                                                                                      21 MB/s |  20 MB     00:00
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                  21 MB/s |  20 MB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                              1/1
  Running scriptlet: rke2-selinux-0.14-1.el8.noarch                                                                                                                                                                               1/3
  Installing       : rke2-selinux-0.14-1.el8.noarch                                                                                                                                                                               1/3
  Running scriptlet: rke2-selinux-0.14-1.el8.noarch                                                                                                                                                                               1/3
  Installing       : rke2-common-1.28.2~rke2r1-0.el8.x86_64                                                                                                                                                                       2/3
  Installing       : rke2-server-1.28.2~rke2r1-0.el8.x86_64                                                                                                                                                                       3/3
  Running scriptlet: rke2-server-1.28.2~rke2r1-0.el8.x86_64                                                                                                                                                                       3/3
  Running scriptlet: rke2-selinux-0.14-1.el8.noarch                                                                                                                                                                               3/3
  Running scriptlet: rke2-server-1.28.2~rke2r1-0.el8.x86_64                                                                                                                                                                       3/3
  Verifying        : rke2-selinux-0.14-1.el8.noarch                                                                                                                                                                               1/3
  Verifying        : rke2-common-1.28.2~rke2r1-0.el8.x86_64                                                                                                                                                                       2/3
  Verifying        : rke2-server-1.28.2~rke2r1-0.el8.x86_64                                                                                                                                                                       3/3

Installed:
  rke2-common-1.28.2~rke2r1-0.el8.x86_64                                         rke2-selinux-0.14-1.el8.noarch                                         rke2-server-1.28.2~rke2r1-0.el8.x86_64

Complete!
# ls -lZ /usr/bin/rke2
-rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 89186504 Sep 18 18:18 /usr/bin/rke2
brownz11 commented 1 year ago

Can confirm this impacts RHEL8 as well. Downgrading to 0.14-1.el8 fixes it.

brownz11 commented 1 year ago

While trying to reproduce I did find if you update container-selinux (for me from version2.205.0-2.module+el8.8.0+18438+15d3aa65. to version 2.205.0-2.module+el8.8.0+19993+47c8ef84), and then reinstall (or upgrade before even attempting to install) rke2-selinux0.15-1 everything is okay. So it may be 0.15-1 needs to have a dependency on a newer version of container-selinux

container-selinux.noarch                      2:2.205.0-2.module+el8.8.0+19993+47c8ef84
rke2-selinux.noarch                           0.15-1.el8

# ls -lZ /usr/bin/rke2
-rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 85604744 Sep 18 17:18 /usr/bin/rke2

It appears updating container-selinux and then reinstalled rke2-selinux will break something and prevent context from being set correctly on some containers. If you've found yourself in a situation where pods are crashing due to SELinux denials, and the labeling seems off, you can purge all the images/snapshots from the node and force them to get repulled and labeled correctly

systemctl stop rke2-server
rke2-killall.sh
cd /var/lib/rancher/rke2/bin/
./containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd &
export CONTAINER_RUNTIME_ENDPOINT=unix:///run/k3s/containerd/containerd.sock
export CONTAINERD_ADDRESS=/run/k3s/containerd/containerd.sock
./crictl rmi -a
#Run this next command until it doesn't error 
./ctr -n k8s.io snapshots ls | cut -d' ' -f 1 | grep -v "KEY" | xargs -i ./ctr -n k8s.io snapshots rm {} || true
./ctr -n k8s.io snapshots ls | cut -d' ' -f 1 | grep -v "KEY" | xargs -i ./ctr -n k8s.io snapshots rm {} || true
./ctr -n k8s.io snapshots ls | cut -d' ' -f 1 | grep -v "KEY" | xargs -i ./ctr -n k8s.io snapshots rm {} || true
fg
[CTRL-C]
skepickle commented 1 year ago

Can confirm that this impacts RHEL8.7 as well.

brownz11 commented 1 year ago

After more digging, there appears to be a conflict between rke2-selinux and older versions of container-selinux.

The RKE2 installer will unload the RKE2 module (due to https://github.com/rancher/rke2/blob/87e1779357832973c26ea33ed636a7abb727c711/install.sh#L563) Try to install rke2-selinux It will fail to build it's module with an error about "Failed to resolve typeattributeset statement at /var/lib/selinux/targeted/tmp/modules/400/rke2/cil:63"

And then goes on to restorecon everything without the RKE2 module loaded, mislabeling everything.

This means the rke2 module never gets loaded back into the system.