search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
27.64k stars 2.32k forks source link

Deployment of k3s worker on Centos7 fails because of SELinux #2162

Closed pgonin closed 4 years ago

pgonin commented 4 years ago

Environmental Info: K3s Version: 1.18.8

Node(s) CPU architecture, OS, and Version: Linux k3s-2 4.14.40-sunxi64 rancher/k3s#180 SMP Mon May 14 23:27:03 CEST 2018 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration: 1 master deployed on raspberry pi 3b / openSUSE MicroOS Linux k3s-1 5.8.0-1-default rancher/k3s#1 SMP Tue Aug 4 07:30:59 UTC 2020 (9bc0044) aarch64 aarch64 aarch64 GNU/Linux

Deploying first worker on pine64 / CentOS7

Describe the bug: Deployment of k3s worker on Centos7 / aarch64 pine64 fails because of SELinux [ERROR] Failed to apply container_runtime_exec_t to /usr/local/bin/k3s

Steps To Reproduce:

# curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=mynodetoken sh -
[ERROR]  Failed to apply container_runtime_exec_t to /usr/local/bin/k3s, please install:
    yum install -y container-selinux selinux-policy-base
    rpm -i https://rpm.rancher.io/k3s-selinux-0.1.1-rc1.el7.noarch.rpm

Additional context / logs: Requested packages are installed and deployment is still failing

# yum install -y container-selinux selinux-policy-base
[...]
Package 2:container-selinux-2.119.2-1.911c772.el7_8.noarch already installed and latest version
Package selinux-policy-targeted-3.13.1-266.el7_8.1.noarch already installed and latest version
Nothing to do
# rpm -i https://rpm.rancher.io/k3s-selinux-0.1.1-rc1.el7.noarch.rpm
warning: /var/tmp/rpm-tmp.CmOYJu: Header V4 RSA/SHA1 Signature, key ID e257814a: NOKEY
# rpm -ql k3s-selinux
/usr/share/selinux/devel/include/contrib/k3s.if
/usr/share/selinux/packages/k3s.pp
# rpm -qi k3s-selinux
Name        : k3s-selinux
Version     : 0.1.1
Release     : rc1.el7
Architecture: noarch
Install Date: Tue 25 Aug 2020 02:01:35 PM UTC
Group       : System Environment/Base
Size        : 83378
License     : ASL 2.0
Signature   : RSA/SHA1, Tue 17 Mar 2020 07:46:01 PM UTC, Key ID 925ea29ae257814a
Source RPM  : k3s-selinux-0.1.1-rc1.el7.src.rpm
Build Date  : Tue 17 Mar 2020 07:45:52 PM UTC
Build Host  : bf7f3e2fbe51
Relocations : (not relocatable)
URL         : http://k3s.io
Summary     : SELinux policy module for k3s
Description :
This package installs and sets up the  SELinux policy security module for k3s.
brandond commented 4 years ago

cc @dweomer

dweomer commented 4 years ago

@pgonin the install script is swallowing the output from:

sudo chcon -u system_u -r object_r -t container_runtime_exec_t /usr/local/bin/k3s

Can you please try that command and provide the output here?

pgonin commented 4 years ago

here it is

[root@k3s-2 ~]# sudo chcon -u system_u -r object_r -t container_runtime_exec_t /usr/local/bin/k3s
chcon: can't apply partial context to unlabeled file ‘/usr/local/bin/k3s’
dweomer commented 4 years ago

sudo restorecon -v /usr/local/bin/k3s && ls -Z /usr/local/bin/k3s ? also what does getenforce return?

pgonin commented 4 years ago 
[root@k3s-2 ~]# restorecon -v /usr/local/bin/k3s && ls -Z /usr/local/bin/k3s 
-rwxr-xr-x root root ?                                /usr/local/bin/k3s
[root@k3s-2 ~]# getenforce
Disabled

and still

[root@k3s-2 ~]# sudo chcon -u system_u -r object_r -t container_runtime_exec_t /usr/local/bin/k3s
chcon: can't apply partial context to unlabeled file ‘/usr/local/bin/k3s’
dweomer commented 4 years ago

Ah, so, $(getenforce) = "Disabled" is something that we should be checking for in the install.sh and bypassing selinux if that is the case. It looks like we are relying on /etc/selinux/config having SELINUX=enforcing which is the likely miscue here. The workaround for you is to fix your /etc/selinux/config. Or, if you WANT SELINUX=enforcing, make sure that the /sys/fs/selinux filesystem is mounted.

I use this script commonly to easily toggle between all three SELinux "modes":

#!/usr/bin/env bash
set -eux -o pipefail

if ! type -p getenforce setenforce &>/dev/null; then
  echo SELinux is Disabled
  exit 0  
fi

case "${SELINUX}" in
  Disabled)
    if mountpoint -q /sys/fs/selinux; then
      setenforce 0
      umount -v /sys/fs/selinux
    fi
    ;;
  Enforcing)
    mountpoint -q /sys/fs/selinux || mount -o rw,relatime -t selinuxfs selinuxfs /sys/fs/selinux
    setenforce 1
    ;;
  Permissive)
    mountpoint -q /sys/fs/selinux || mount -o rw,relatime -t selinuxfs selinuxfs /sys/fs/selinux
    setenforce 0
    ;;
  *)
    echo "SELinux mode not supported: ${SELINUX}" >&2
    exit 1  
    ;;
esac

echo SELinux is $(getenforce)
dweomer commented 4 years ago

Ah, so, $(getenforce) = "Disabled" is something that we should be checking for in the install.sh and bypassing selinux if that is the case. It looks like we are relying on /etc/selinux/config having SELINUX=enforcing which is the likely miscue here. The workaround for you is to fix your /etc/selinux/config. Or, if you WANT SELINUX=enforcing, make sure that the /sys/fs/selinux filesystem is mounted.

I use this script commonly to easily toggle between all three SELinux "modes":

#!/usr/bin/env bash
set -eux -o pipefail

if ! type -p getenforce setenforce &>/dev/null; then
  echo SELinux is Disabled
  exit 0  
fi

case "${SELINUX}" in
  Disabled)
    if mountpoint -q /sys/fs/selinux; then
      setenforce 0
      umount -v /sys/fs/selinux
    fi
    ;;
  Enforcing)
    mountpoint -q /sys/fs/selinux || mount -o rw,relatime -t selinuxfs selinuxfs /sys/fs/selinux
    setenforce 1
    ;;
  Permissive)
    mountpoint -q /sys/fs/selinux || mount -o rw,relatime -t selinuxfs selinuxfs /sys/fs/selinux
    setenforce 0
    ;;
  *)
    echo "SELinux mode not supported: ${SELINUX}" >&2
    exit 1  
    ;;
esac

echo SELinux is $(getenforce)

The thing is, centos usually has selinux enforcing enabled by default. I imagine it is disabled for rpi3 to minimize writes to an fs commonly backed by an sd-card.

pgonin commented 4 years ago

Indeed I disabled SELinux in /etc/selinux/config with SELINUX=disabled and k3s configuration & start is now ok

I actually used a CentOS7 image with Armbian kernel from https://project31.github.io/pine64/ So the issue might be specific to this image. I could give a try to a regular CentOS image on rpi to check if it has the same issue.

But from my perspective, issue can be closed. Thanks a lot !

bertreyking commented 1 year ago

I disabled SELinux in /etc/selinux/config with SELINUX=disabled and k3s configuration & start is now ok