search

rancher / rke2-selinux

RKE2 selinux + RPM packaging for selinux
Apache License 2.0
21 stars 21 forks source link

Denied watches on `/var/lib/rancher/rke2/server/tls/` files #31

Open derekmwright opened 1 year ago

derekmwright commented 1 year ago

We are seeing the following denials in our logs for certificates/files in /var/lib/rancher/rke2/server/tls/. After doing some poking around it seems newer kernels may explicitly need watch permissions set?

Jan 25 21:57:02 ip-192-168-64-212 audit[1936]: AVC avc:  denied  { watch } for  pid=1936 comm="kube-apiserver" path="/var/lib/rancher/rke2/server/tls/request-header-ca.crt" dev="nvme0n1p4" ino=73400475 scontext=system_u:system_r:rke2_service_t:s0:c112,c840 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
Jan 25 21:57:02 ip-192-168-64-212 audit[1936]: AVC avc:  denied  { watch } for  pid=1936 comm="kube-apiserver" path="/var/lib/rancher/rke2/server/tls/request-header-ca.crt" dev="nvme0n1p4" ino=73400475 scontext=system_u:system_r:rke2_service_t:s0:c112,c840 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
Jan 25 21:57:02 ip-192-168-64-212 audit[1936]: AVC avc:  denied  { watch } for  pid=1936 comm="kube-apiserver" path="/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt" dev="nvme0n1p4" ino=73400472 scontext=system_u:system_r:rke2_service_t:s0:c112,c840 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
Jan 25 21:57:15 ip-192-168-64-212 audit[2081]: AVC avc:  denied  { watch } for  pid=2081 comm="kube-controller" path="/var/lib/rancher/rke2/server/tls/server-ca.crt" dev="nvme0n1p4" ino=73400470 scontext=system_u:system_r:rke2_service_t:s0:c513,c655 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
Jan 25 21:57:15 ip-192-168-64-212 audit[2081]: AVC avc:  denied  { watch } for  pid=2081 comm="kube-controller" path="/var/lib/rancher/rke2/server/tls/client-ca.crt" dev="nvme0n1p4" ino=73400453 scontext=system_u:system_r:rke2_service_t:s0:c513,c655 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
Jan 25 21:57:15 ip-192-168-64-212 audit[2081]: AVC avc:  denied  { watch } for  pid=2081 comm="kube-controller" path="/var/lib/rancher/rke2/server/tls/client-ca.crt" dev="nvme0n1p4" ino=73400453 scontext=system_u:system_r:rke2_service_t:s0:c513,c655 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
Jan 25 21:57:15 ip-192-168-64-212 audit[2081]: AVC avc:  denied  { watch } for  pid=2081 comm="kube-controller" path="/var/lib/rancher/rke2/server/tls/server-ca.crt" dev="nvme0n1p4" ino=73400470 scontext=system_u:system_r:rke2_service_t:s0:c513,c655 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
Jan 25 21:57:15 ip-192-168-64-212 kernel: audit: type=1400 audit(1674683835.676:1145): avc:  denied  { watch } for  pid=2081 comm="kube-controller" path="/var/lib/rancher/rke2/server/tls/server-ca.crt" dev="nvme0n1p4" ino=73400470 scontext=system_u:system_r:rke2_service_t:s0:c513,c655 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
Jan 25 21:57:15 ip-192-168-64-212 kernel: audit: type=1400 audit(1674683835.676:1146): avc:  denied  { watch } for  pid=2081 comm="kube-controller" path="/var/lib/rancher/rke2/server/tls/client-ca.crt" dev="nvme0n1p4" ino=73400453 scontext=system_u:system_r:rke2_service_t:s0:c513,c655 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
Jan 25 21:57:15 ip-192-168-64-212 kernel: audit: type=1400 audit(1674683835.676:1147): avc:  denied  { watch } for  pid=2081 comm="kube-controller" path="/var/lib/rancher/rke2/server/tls/client-ca.crt" dev="nvme0n1p4" ino=73400453 scontext=system_u:system_r:rke2_service_t:s0:c513,c655 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
Jan 25 21:57:15 ip-192-168-64-212 kernel: audit: type=1400 audit(1674683835.676:1148): avc:  denied  { watch } for  pid=2081 comm="kube-controller" path="/var/lib/rancher/rke2/server/tls/server-ca.crt" dev="nvme0n1p4" ino=73400470 scontext=system_u:system_r:rke2_service_t:s0:c513,c655 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0

This is a Fedora CoreOS system:

# cat /etc/redhat-release
Fedora release 37 (Thirty Seven)
# uname -a
Linux ip-192-168-64-212 6.0.7-301.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 4 18:35:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
dougkirkley commented 1 year ago

I'm also running into this issue on Fedora CoreOS 37. Any ideas?

jcmontigny commented 1 year ago

I have the same issue in Rocky Linux 9.

The process correctly runs with the associated context and the cluster also runs correctly : 

# ps -e --context | grep kube-controller
   5233 system_u:system_r:container_t:s0:c2,c434 /usr/bin/kube-controllers
   5386 system_u:system_r:rke2_service_t:s0:c196,c520 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true
  97660 unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 grep --color=auto kube-controller

Maybe there is feature to rotate certificates automatically without requiring a controlplane restart, so perhaps it depends on watching the file and the permission was simply not added to the rules...

jhoelzel commented 1 year ago

Im also experiencing this using the latest MicroOS and RKE2 versions

48 potentially solves this

galal-hussein commented 1 year ago

please refer to my answer on the same issue https://github.com/rancher/rke2-selinux/issues/47#issuecomment-1670070124

jhoelzel commented 1 year ago

please refer to my answer on the same issue #47 (comment)

If i am not mistaken then MicroOS should be able to use this too:

microos-s-1vcpu-1gb-nyc3-01:~ # cat /proc/version
Linux version 6.4.8-1-default (geeko@buildhost) (gcc (SUSE Linux) 13.1.1 20230720 [revision 9aac37ab8a7b919a89c6d64bc7107a8436996e93], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.40.0.20230412-5) #1 SMP PREEMPT_DYNAMIC Thu Aug  3 13:32:15 UTC 2023 (c24ac79)
galal-hussein commented 1 year ago

@jhoelzel Thats true but the microos package still being used in SLES, so it still might case a conflict

jhoelzel commented 1 year ago

@jhoelzel Thats true but the microos package still being used in SLES, so it still might case a conflict

So a fix will not see the light of day for quite some time and i should roll my own?

jhoelzel commented 1 year ago

I have for now solved this witha temporary custom policy like this:

rke2-selinux-workaround.te:

module rke2-selinux-workaround 1.0;

require {
        type var_lib_t;
        type container_var_lib_t;
        type rke2_service_t;
        class dir { write add_name};
        class file { watch };
}

#============= rke2_service_t ==============
allow rke2_service_t container_var_lib_t:file watch;
allow rke2_service_t var_lib_t:dir write;
allow rke2_service_t var_lib_t:dir add_name;

and you can apply this on MicroOS like this (in a transactional update of course):

        cd /etc/rancher/rke2/semodule
        checkmodule -M -m -o rke2-selinux-workaround.mod rke2-selinux-workaround.te
        semodule_package -o  rke2-selinux-workaround.pp -m  rke2-selinux-workaround.mod
        semodule -i  rke2-selinux-workaround.pp
galal-hussein commented 1 year ago

@jhoelzel I am waiting for official reply from sle micro distro to confirm the current kernel shipped with sle micro to apply the watch fix to it.