iscsiadm missing from Selinux policy

[clemenko]

This is an extension of #36. with versions : Rocky 9.1, and 9.2 rke2 - v1.24.14+rke2r1 container-selinux.noarch - 3:2.205.0-1.el9_2 rke2-selinux.noarch - 0.12-1.el9

Before the container-selinux update to 205 Longhorn was working. Now, not so much.

Here are the recreation steps:

yum update -y
yum install -y nfs-utils cryptsetup iscsi-initiator-utils; systemctl enable --now iscsid
curl -sfL https://get.rke2.io | INSTALL_RKE2_CHANNEL=v1.24 sh - && systemctl enable --now rke2-server.service

echo "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml PATH=$PATH:/var/lib/rancher/rke2/bin" >> ~/.bashrc
ln -s /var/run/k3s/containerd/containerd.sock /var/run/containerd/containerd.sock
source ~/.bashrc

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm repo add longhorn https://charts.longhorn.io --force-update
helm upgrade -i longhorn  longhorn/longhorn -n longhorn-system --create-namespace --set ingress.enabled=true --set ingress.host=<IP>.nip.io --set default.storageMinimalAvailablePercentage=25 --set default.storageOverProvisioningPercentage=200 --set default.persistence.defaultDataLocality=best-effort

kubectl apply -f https://raw.githubusercontent.com/clemenko/k8s_yaml/master/flask_simple_nginx.yml

kubectl get pod -n flask

The redis pod is stuck creating because the isciadm is blocked by a selinux policy.

Here is the error:

Jun  1 01:38:27 flux setroubleshoot[77955]: SELinux is preventing /usr/sbin/iscsiadm from using the dac_override capability. For complete SELinux messages run: sealert -l adcd3346-8687-4402-a479-bdd758948d4e
Jun  1 01:38:27 flux setroubleshoot[77955]: SELinux is preventing /usr/sbin/iscsiadm from using the dac_override capability.#012#012*****  Plugin dac_override (91.4 confidence) suggests   **********************#012#012If you want to help identify if domain needs this access or you have a file with the wrong permissions on your system#012Then turn on full auditing to get path information about the offending file and generate the error again.#012Do#012#012Turn on full auditing#012# auditctl -w /etc/shadow -p w#012Try to recreate AVC. Then execute#012# ausearch -m avc -ts recent#012If you see PATH record check ownership/permissions on file, and fix it,#012otherwise report as a bugzilla.#012#012*****  Plugin catchall (9.59 confidence) suggests   **************************#012#012If you believe that iscsiadm should have the dac_override capability by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'iscsiadm' --raw | audit2allow -M my-iscsiadm#012# semodule -X 300 -i my-iscsiadm.pp#012
Jun  1 01:38:27 flux setroubleshoot[77955]: SELinux is preventing /usr/sbin/iscsiadm from using the dac_override capability. For complete SELinux messages run: sealert -l adcd3346-8687-4402-a479-bdd758948d4e
Jun  1 01:38:27 flux setroubleshoot[77955]: SELinux is preventing /usr/sbin/iscsiadm from using the dac_override capability.#012#012*****  Plugin dac_override (91.4 confidence) suggests   **********************#012#012If you want to help identify if domain needs this access or you have a file with the wrong permissions on your system#012Then turn on full auditing to get path information about the offending file and generate the error again.#012Do#012#012Turn on full auditing#012# auditctl -w /etc/shadow -p w#012Try to recreate AVC. Then execute#012# ausearch -m avc -ts recent#012If you see PATH record check ownership/permissions on file, and fix it,#012otherwise report as a bugzilla.#012#012*****  Plugin catchall (9.59 confidence) suggests   **************************#012#012If you believe that iscsiadm should have the dac_override capability by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'iscsiadm' --raw | audit2allow -M my-iscsiadm#012# semodule -X 300 -i my-iscsiadm.pp#012

[clemenko]

I was able to get it working with

ausearch -c 'iscsiadm' --raw | audit2allow -M my-iscsiadm;  semodule -X 300 -i my-iscsiadm.pp

Here are the polices that made this work :

[root@rke3 ~]# cat my-iscsiadm.pp 
??|???|?SE Linux Module
                      my-iscsiadm1.0@
capability
          dac_overridobject_r@@@@iscsid_t@@@@@@@@@@@@@@@@@@@@@@@@@

and

[root@rke3 ~]# cat my-iscsiadm.te 

module my-iscsiadm 1.0;

require {
    type iscsid_t;
    class capability dac_override;
}

#============= iscsid_t ==============

#!!!! This avc is allowed in the current policy
allow iscsid_t self:capability dac_override;

Can we update the rke2-selinux policy with this?

[zackbradys]

+1 ... seeing this issue as well on RHEL and Rocky Linux.

[clemenko]

More info

[root@rke3 ~]# ls -aslZ /var/lib/longhorn/replicas/pvc-5d74ae4e-a994-4127-9573-3f58e536ba89-87b4e2f3/
total 35900
    0 drwx------. 2 root root system_u:object_r:container_var_lib_t:s0       108 Jun  1 12:28 .
    0 drwxr-xr-x. 3 root root system_u:object_r:container_var_lib_t:s0        63 Jun  1 12:27 ..
    4 -rw-------. 1 root root system_u:object_r:container_var_lib_t:s0      4096 Jun  1 12:29 revision.counter
35888 -rw-r--r--. 1 root root system_u:object_r:container_var_lib_t:s0 262144000 Jun  1 12:29 volume-head-000.img
    4 -rw-r--r--. 1 root root system_u:object_r:container_var_lib_t:s0       126 Jun  1 12:27 volume-head-000.img.meta
    4 -rw-r--r--. 1 root root system_u:object_r:container_var_lib_t:s0       141 Jun  1 12:28 volume.meta
[root@rke3 ~]# ps -efZ | grep -i iscsi
system_u:system_r:iscsid_t:s0   root        5575       1  0 12:20 ?        00:00:00 /usr/sbin/iscsid -f
system_u:system_r:kernel_t:s0   root        5618       2  0 12:20 ?        00:00:00 [iscsi_conn_clea]
system_u:system_r:kernel_t:s0   root       30342       2  0 12:28 ?        00:00:00 [iscsi_q_3]
system_u:system_r:kernel_t:s0   root       30345       2  0 12:28 ?        00:00:00 [iscsi_ctrl_3:1]
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 35023 27675  0 12:31 pts/0 00:00:00 grep --color=auto -i iscsi

the process does not have the same context as the directory. this does fix things

ausearch -c 'iscsiadm' --raw | audit2allow -M my-iscsiadm;  semodule -X 300 -i my-iscsiadm.pp

[clemenko]

I wrote a patch that works https://github.com/clemenko/selinux_longhorn_patch Would be nice to include this into the rke2-selinux policies.

[Martin-Weiss]

@clemenko - wouldn´t this patch not make more sense on the OS side of things where iscsiadm is delivered via RPM? Does the OS deliver any selinux policy for iSCSI?

[clemenko]

No. This is due to the selinux policies that are applied by container-selinux and rke2-selinux. The directory is tagged with system_u:object_r:container_var_lib_t:s0. and the longhorn pod uses the system_u:system_r:iscsid_t:s0 tag. the mismatch is causing the issue. Yes this is potentially a longhorn issue. But they have nothing to do with selinux. This makes the rke2-linux the best place for it. Also creating a good foundation for all our product should be the goal of rke2 in general.

[clemenko]

Update - chcon -t container_file_t /usr/sbin/iscsiadm works as a work around right now. more to come.

[ejweber]

No. This is due to the selinux policies that are applied by container-selinux and rke2-selinux. The directory is tagged with system_u:object_r:container_var_lib_t:s0. and the longhorn pod uses the system_u:system_r:iscsid_t:s0 tag. the mismatch is causing the issue. Yes this is potentially a longhorn issue. But they have nothing to do with selinux. This makes the rke2-linux the best place for it. Also creating a good foundation for all our product should be the goal of rke2 in general.

Can you clarify this a bit @clemenko. My research doesn't indicate that any sort of context mismatch under the /var/lib/longhorn directory is causing problems. This seems to be the result of an open-iscsi bug that is not fully patched in iscsi-initiator-utils on RHEL/Rocky requiring the dac_override permission for directories under /var/lib/iscsi.

https://github.com/longhorn/longhorn/issues/5627#issuecomment-1573959980

[clemenko]

chcon -t container_file_t /usr/sbin/iscsiadm fixes the issue.

here

#notworking : 
[root@rke2 ~]# ls -aslZ /usr/sbin/iscsiadm 
408 -rwxr-xr-x. 1 root root system_u:object_r:iscsid_exec_t:s0 413920 Oct 31  2022 /usr/sbin/iscsiadm

#working : 
[root@flux ~]# ls -aslZ /usr/sbin/iscsiadm 
408 -rwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0 413920 Oct 31  2022 /usr/sbin/iscsiadm

I was wrong, this does not hit /var/lib/longhorn.

[ejweber]

Testing according to recreate steps at https://github.com/longhorn/longhorn/issues/5627#issuecomment-1573874544 except with curl https://get.rke2.io/ | INSTALL_RKE2_CHANNEL=testing sh:

Container SELinux and its dependencies are up-to-date (i.e. they are the versions that were causing Longhorn-related issues).

[root@selinux-recreate-rocky-9-3 ~]# yum list installed | grep 'container-selinux\|selinux-policy'
container-selinux.noarch            3:2.205.0-1.el9_2           @appstream
selinux-policy.noarch               38.1.11-2.el9_2.2           @baseos
selinux-policy-targeted.noarch      38.1.11-2.el9_2.2           @baseos

There is a rule in place to transition from spc_t to iscsid_t. Nothing is different here. Just checking.

[root@selinux-recreate-rocky-9-3 ~]# sesearch -T -s spc_t | grep iscsid
type_transition spc_t iscsid_exec_t:process iscsid_t;

iscsid_t has dac_override.

[root@selinux-recreate-rocky-9-3 ~]# sesearch -A -s iscsid_t -c capability -p dac_override
allow iscsid_t iscsid_t:capability { dac_override dac_read_search ipc_lock net_admin net_raw sys_admin sys_module sys_nice sys_resource };

After Longhorn is installed, volumes attach as expected and none of the associated error messages are present. (The volume is degraded because I don't have enough nodes to schedule its remaining replicas.)

[root@selinux-recreate-rocky-9-3 longhorn]# kubectl apply -f examples/pod_with_pvc.yaml
persistentvolumeclaim/longhorn-volv-pvc created
pod/volume-test created

[root@selinux-recreate-rocky-9-3 longhorn]# kubectl get -n longhorn-system volume
NAME                                       STATE      ROBUSTNESS   SCHEDULED   SIZE         NODE                         AGE
pvc-8759a6ae-0dee-4bc1-ab6d-2ff3556fda52   attached   degraded                 2147483648   selinux-recreate-rocky-9-3   26s

[root@selinux-recreate-rocky-9-3 longhorn]# kubectl logs -n longhorn-system instance-manager-e-644507d431ae9ad662894414841b08c6 | grep "Failed to discover"
# Empty.

[root@selinux-recreate-rocky-9-3 longhorn]# ausearch -m AVC -ts recent | grep iscsiadm
# Empty.

Looks good to me!

[zackbradys]

In accordance with @galal-hussein latest merge (#45)... verified and tested with curl https://get.rke2.io/ | INSTALL_RKE2_CHANNEL=testing INSTALL_RKE2_VERSION=v1.24.14+rke2r1 sh on Rocky Linux 9.2:

Verify OS Release:

[root@rke3 rke2-artifacts]# cat /etc/os-release 
NAME="Rocky Linux"
VERSION="9.2 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.2 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.2"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"

Verify Packages:

[zackbradys@rke3 ~]$ yum list installed | grep 'container-selinux\|selinux-policy'
container-selinux.noarch                 3:2.205.0-1.el9_2            @appstream
selinux-policy.noarch                    38.1.11-2.el9_2.2            @baseos   
selinux-policy-targeted.noarch           38.1.11-2.el9_2.2            @baseos  

Verify RKE2:

[root@rke3 ~]# kubectl get nodes -o wide
NAME   STATUS   ROLES                       AGE   VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                      KERNEL-VERSION                     CONTAINER-RUNTIME
rke3   Ready    control-plane,etcd,master   14m   v1.24.14+rke2r1   10.0.0.20     <none>        Rocky Linux 9.2 (Blue Onyx)   5.14.0-162.12.1.el9_1.0.2.x86_64   containerd://1.7.1-k3s1

Verify DAC_OVERRIDE:

[root@rke3 zackbradys]# sesearch -A -s iscsid_t -c capability -p dac_override
allow iscsid_t iscsid_t:capability { dac_override dac_read_search ipc_lock net_admin net_raw sys_admin sys_module sys_nice sys_resource };

Verify Longhorn Volumes:

[root@rke3 certs]# kubectl get -n longhorn-system volume
NAME                                       STATE      ROBUSTNESS   SCHEDULED   SIZE         NODE   AGE
pvc-46edd91d-3187-4c3c-8b7a-410aff0a2637   attached   degraded                 1073741824   rke3   25s

[clemenko]

Confirmed working with the release of 0.14-1.el9. Thanks team!