apply_k3s_selinux fails on MicroOS snapshot of 2023-01-23

[valkenburg-prevue-ch]

The script apply_k3s_selinux = ["/sbin/semodule -v -i /usr/share/selinux/packages/k3s.pp"] fails as of this morning, with the following output:

k3s-agent-large-hel1-djs:~ # /sbin/semodule -v -i /usr/share/selinux/packages/k3s.pp
Attempting to install module '/usr/share/selinux/packages/k3s.pp':
Ok: return value of 0.
Committing changes:
Conflicting name type transition rules
Binary policy creation failed at /var/lib/selinux/targeted/tmp/modules/400/k3s/cil:135
Failed to generate binary
/sbin/semodule:  Failed!

I have the strong suspicion that something changed in microos, as the image has been updated yesterday:

Does anyone know how to work around this?

[valkenburg-prevue-ch]

If I make this modification in locals.tf,

  # apply_k3s_selinux = ["/sbin/semodule -v -i /usr/share/selinux/packages/k3s.pp"]
  apply_k3s_selinux = ["echo 'Hello unsafe world!'"]

I confirm that the cluster comes up. However, this made me notice an error message which is probably leading closer to the issue:

The following NEW package is going to be installed:
k3s-selinux
...
Conflicting name type transition rules
Binary policy creation failed at /var/lib/selinux/targeted/tmp/modules/200/k3s/cil:135
Failed to generate binary
/usr/sbin/semodule:  Failed!

However, this message does not block the rest of the installation, it just continues.

[thebearingedge]

I noticed this just a few minutes ago. I'm searching the repository history but still haven't quite grasped the rationale for skipping the automatic selinux installation that comes with the k3s installer.

I haven't learned the consequences yet, but letting the k3s install script include it produces no error.

Edit: oh maybe it's just to allow modification to the configurations before the k3s service is started?

[aleksasiriski]

Same thing here, guess something changed outside of this script (either MicroOS or K3S script) since the latest version worked 2 days ago flawlessly.

[mysticaltech]

We probably just need to update the k3s SELinux RPM! On it.

[mysticaltech]

@valkenburg-prevue-ch You are right something changed with MicroOS, breaking the k3s-SELinux install.

[mysticaltech]

I noticed this just a few minutes ago. I'm searching the repository history but still haven't quite grasped the rationale for skipping the automatic selinux installation that comes with the k3s installer.

I haven't learned the consequences yet, but letting the k3s install script include it produces no error.

Edit: oh maybe it's just to allow modification to the configurations before the k3s service is started?

@thebearingedge In the current state of things, this would have failed too. We just install it manually and apply the policy ourselves because MicroOS installs all new packages to a new Snapshot, and requires a reboot. We remove the need for that extra reboot, but installing the package manually before k3s is installed.

[mysticaltech]

If I make this modification in locals.tf,

  # apply_k3s_selinux = ["/sbin/semodule -v -i /usr/share/selinux/packages/k3s.pp"]
  apply_k3s_selinux = ["echo 'Hello unsafe world!'"]

I confirm that the cluster comes up. However, this made me notice an error message which is probably leading closer to the issue:

The following NEW package is going to be installed:
k3s-selinux
...
Conflicting name type transition rules
Binary policy creation failed at /var/lib/selinux/targeted/tmp/modules/200/k3s/cil:135
Failed to generate binary
/usr/sbin/semodule:  Failed!

However, this message does not block the rest of the installation, it just continues.

@valkenburg-prevue-ch This basically disables k3s-selinux. It a neat trick but not ideal.

I have signaled our problem on the k3s-selinux repo. It turns out even folks on Fedora CoreOS are having the exact same error! So not specific to MicroOS.

[mysticaltech]

Ok, the solution has been found, thanks to the tips given by the maintainer of k3s-selinux.

It required a downgrade of container-selinux, and he is working on fix to support newer versions! 🤞

[valkenburg-prevue-ch]

Oh nooooo! The fix is broken today. The line:

transactional-update shell <<< "zypper install -y --oldpackage https://download.opensuse.org/tumbleweed/repo/oss/noarch/container-selinux-2.188.0-2.1.noarch.rpm && zypper addlock container-selinux"

fails because https://download.opensuse.org/tumbleweed/repo/oss/noarch/container-selinux-2.188.0-2.1.noarch.rpm no longer exists!

Reopening the issue.

[valkenburg-prevue-ch]

Ok, I missed one more commit of the master branch, it seems fixed there already. Thanks @mysticaltech and @aleksasiriski .

[mysticaltech]

Apologies folks, I screwed up the link. Thanks for fixing it @aleksasiriski 🙏

[mariusleu]

This error hits again on a fresh MicroOS image:

module.kube-hetzner.null_resource.first_control_plane (remote-exec): + curl -sfL https://get.k3s.io
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + INSTALL_K3S_SKIP_START=true
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + INSTALL_K3S_SKIP_SELINUX_RPM=true
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + INSTALL_K3S_CHANNEL=stable
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + INSTALL_K3S_EXEC='server '
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + sh -
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Finding release for channel stable
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Using v1.30.3+k3s1 as release
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.30.3+k3s1/sha256sum-amd64.txt
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.30.3+k3s1/k3s
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Verifying binary download
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Installing k3s to /usr/local/bin/k3s
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Skipping installation of SELinux RPM
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, already exists
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Skipping /usr/local/bin/crictl symlink to k3s, already exists
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Skipping /usr/local/bin/ctr symlink to k3s, already exists
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  systemd: Enabling k3s unit
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Created symlink '/etc/systemd/system/multi-user.target.wants/k3s.service' → '/etc/systemd/system/k3s.service'.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + /sbin/semodule -v -i /usr/share/selinux/packages/k3s.pp
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Attempting to install module '/usr/share/selinux/packages/k3s.pp':
module.kube-hetzner.null_resource.first_control_plane (remote-exec): libsemanage.map_compressed_file: Unable to open /usr/share/selinux/packages/k3s.pp
module.kube-hetzner.null_resource.first_control_plane (remote-exec):  (No such file or directory).
module.kube-hetzner.null_resource.first_control_plane (remote-exec): libsemanage.semanage_direct_install_file: Unable to read file /usr/share/selinux/packages/k3s.pp
module.kube-hetzner.null_resource.first_control_plane (remote-exec):  (No such file or directory).
module.kube-hetzner.null_resource.first_control_plane (remote-exec): /sbin/semodule:  Failed on /usr/share/selinux/packages/k3s.pp!
╷
│ Error: remote-exec provisioner error
│ 
│   with module.kube-hetzner.null_resource.first_control_plane,
│   on .terraform/modules/kube-hetzner/init.tf line 70, in resource "null_resource" "first_control_plane":
│   70:   provisioner "remote-exec" {
│ 
│ error executing "/tmp/terraform_102051961.sh": Process exited with status 1

Anyone else experiencing this?

[geeksven-dev]

This error hits again on a fresh MicroOS image:

module.kube-hetzner.null_resource.first_control_plane (remote-exec): + curl -sfL https://get.k3s.io
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + INSTALL_K3S_SKIP_START=true
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + INSTALL_K3S_SKIP_SELINUX_RPM=true
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + INSTALL_K3S_CHANNEL=stable
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + INSTALL_K3S_EXEC='server '
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + sh -
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Finding release for channel stable
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Using v1.30.3+k3s1 as release
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.30.3+k3s1/sha256sum-amd64.txt
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.30.3+k3s1/k3s
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Verifying binary download
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Installing k3s to /usr/local/bin/k3s
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Skipping installation of SELinux RPM
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, already exists
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Skipping /usr/local/bin/crictl symlink to k3s, already exists
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Skipping /usr/local/bin/ctr symlink to k3s, already exists
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  systemd: Enabling k3s unit
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Created symlink '/etc/systemd/system/multi-user.target.wants/k3s.service' → '/etc/systemd/system/k3s.service'.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + /sbin/semodule -v -i /usr/share/selinux/packages/k3s.pp
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Attempting to install module '/usr/share/selinux/packages/k3s.pp':
module.kube-hetzner.null_resource.first_control_plane (remote-exec): libsemanage.map_compressed_file: Unable to open /usr/share/selinux/packages/k3s.pp
module.kube-hetzner.null_resource.first_control_plane (remote-exec):  (No such file or directory).
module.kube-hetzner.null_resource.first_control_plane (remote-exec): libsemanage.semanage_direct_install_file: Unable to read file /usr/share/selinux/packages/k3s.pp
module.kube-hetzner.null_resource.first_control_plane (remote-exec):  (No such file or directory).
module.kube-hetzner.null_resource.first_control_plane (remote-exec): /sbin/semodule:  Failed on /usr/share/selinux/packages/k3s.pp!
╷
│ Error: remote-exec provisioner error
│ 
│   with module.kube-hetzner.null_resource.first_control_plane,
│   on .terraform/modules/kube-hetzner/init.tf line 70, in resource "null_resource" "first_control_plane":
│   70:   provisioner "remote-exec" {
│ 
│ error executing "/tmp/terraform_102051961.sh": Process exited with status 1

Anyone else experiencing this?

Maybe you also have a spoiled known_hosts and this would be the solution too https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/1481

[mysticaltech]

I will have a look today