search

rancher / rke2-selinux

RKE2 selinux + RPM packaging for selinux
Apache License 2.0
21 stars 21 forks source link

rke2 fails to start with SELinux enabled #65

Closed rdoxenham closed 9 months ago

rdoxenham commented 10 months ago

Using the latest rke2-selinux (rke2-selinux-0.17-1.slemicro.noarch) RPM on SLE Micro 5.5, I am having problems starting RKE2. It seems that the SELinux policy is preventing install-cnis.sh from running and hence pods cannot be started. In the rke2-server logs we see outputs such as this, which loops:

time="2024-01-23T17:23:27.639181170Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:helm-install-endpoint-copier-operator-jmgc9,Uid:01063222-91bd-434c-92b0-ad358b89df6b,Namespace:endpoint-copier-operator,Attempt:0,} failed, error" error="failed to setup network for sandbox \"7a354c0130fe903e1303536d22e78d8bc755d5110544a599eccd35a6a9aaccf9\": plugin type=\"portmap\" failed (add): failed to find plugin \"portmap\" in path [/opt/cni/bin]"
time="2024-01-23T17:23:28.215727832Z" level=info msg="StopPodSandbox for \"7a354c0130fe903e1303536d22e78d8bc755d5110544a599eccd35a6a9aaccf9\""
time="2024-01-23T17:23:28.215959081Z" level=info msg="Ensure that sandbox 7a354c0130fe903e1303536d22e78d8bc755d5110544a599eccd35a6a9aaccf9 in task-service has been cleanup successfully"
time="2024-01-23T17:23:28.218712393Z" level=error msg="StopPodSandbox for \"7a354c0130fe903e1303536d22e78d8bc755d5110544a599eccd35a6a9aaccf9\" failed" error="failed to destroy network for sandbox \"7a354c0130fe903e1303536d22e78d8bc755d5110544a599eccd35a6a9aaccf9\": plugin type=\"portmap\" failed (delete): failed to find plugin \"portmap\" in path [/opt/cni/bin]"
time="2024-01-23T17:23:28.411272279Z" level=error msg="Failed to destroy network for sandbox \"5dbfb6e9819aacd6338e4fa4ed13e4cdabe79ff3c7040c2e21dff52f55e03c16\"" error="plugin type=\"portmap\" failed (delete): failed to find plugin \"portmap\" in path [/opt/cni/bin]"
time="2024-01-23T17:23:28.411865207Z" level=error msg="encountered an error cleaning up failed sandbox \"5dbfb6e9819aacd6338e4fa4ed13e4cdabe79ff3c7040c2e21dff52f55e03c16\", marking sandbox state as SANDBOX_UNKNOWN" error="plugin type=\"portmap\" failed (delete): failed to find plugin \"portmap\" in path [/opt/cni/bin]"

Here's the rke2 configuration:

cni: cilium
selinux: true
server: https://192.168.122.100:9345
tls-san:
    - 192.168.122.100
    - api.cluster01.edge.rdo.wales
token: foobar

We can workaround this by using ausearch and audit2allow to figure out a policy:

node1:~ # ausearch -m AVC
----
time->Wed Jan 24 07:48:54 2024
type=AVC msg=audit(1706082534.718:156): avc:  denied  { write } for  pid=4158 comm="install-cnis.sh" name="bin" dev="vda3" ino=277 scontext=system_u:system_r:container_t:s0:c554,c586 tcontext=system_u:object_r:usr_t:s0 tclass=dir permissive=1
----
time->Wed Jan 24 07:48:54 2024
type=AVC msg=audit(1706082534.730:157): avc:  denied  { add_name } for  pid=4174 comm="cp" name="dummy" scontext=system_u:system_r:container_t:s0:c554,c586 tcontext=system_u:object_r:usr_t:s0 tclass=dir permissive=1
----
time->Wed Jan 24 07:48:54 2024
type=AVC msg=audit(1706082534.730:158): avc:  denied  { create } for  pid=4174 comm="cp" name="dummy" scontext=system_u:system_r:container_t:s0:c554,c586 tcontext=system_u:object_r:usr_t:s0 tclass=file permissive=1
----
time->Wed Jan 24 07:48:54 2024
type=AVC msg=audit(1706082534.730:159): avc:  denied  { write } for  pid=4174 comm="cp" path="/host/opt/cni/bin/dummy" dev="vda3" ino=278 scontext=system_u:system_r:container_t:s0:c554,c586 tcontext=system_u:object_r:usr_t:s0 tclass=file permissive=1
----
time->Wed Jan 24 07:49:24 2024
type=AVC msg=audit(1706082564.027:1071): avc:  denied  { associate } for  pid=8198 comm="nginx" name="2" scontext=system_u:object_r:container_t:s0:c347,c983 tcontext=system_u:object_r:proc_t:s0 tclass=filesystem permissive=1
node1:~ # ausearch -m AVC | audit2allow

#============= container_t ==============
allow container_t proc_t:filesystem associate;
allow container_t usr_t:dir { add_name write };
allow container_t usr_t:file { create write };

I've tested this and it allows RKE2+SELinux (with selinux: true) to work just fine.

This might be related to https://github.com/rancher/rke2-selinux/issues/62

rdoxenham commented 9 months ago

We identified that this is caused by the rke2-selinux %post install script in the RPM failing to create the required directories during Combustion. Therefore we worked around this with a simple fix in our image building tools https://github.com/suse-edge/edge-image-builder/pull/157. As this is seemingly specific to deployment during Combustion, I think we can close this.