[BUG] cattle-cluster-agent does not start afert apply cisProfile: cis

[jisnardo]

Rancher Server Setup

• Rancher version: 2.9.1
• Installation option (Docker install/Helm Chart): Helm Chart
  • If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE2
• Proxy/Cert Details: n/a

Information about the Cluster

• Kubernetes version: v1.30.4+rke2r1

• Cluster Type (Local/Downstream): Downstream

  • If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): Imported

• Hosted = on premise

• Infrastructure Provider = vSphere

User Information

• What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom) : Admin
  • If custom, define the set of permissions: n/a

Describe the bug

FailedCreate    ReplicaSet cattle-cluster-agent-5745cc5df8    Error creating: pods "cattle-cluster-agent-5745cc5df8-sgbhm" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "cluster-register" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "cluster-register" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "cluster-register" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "cluster-register" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")    cattle-cluster-agent-5745cc5df8.17f87b2a62c83032

To Reproduce

• Deploy a rke2 cluster with turtles and fleet.
• Apply cisProfile: cis to RKE2ControlPlane and RKE2ConfigTemplate.

Result

Events started to appear at Rancher.

Expected Result

No impact, hardening applied.

Screenshots

n/a

Additional context

• rancher turtles v1.11.0
• core cluster-api v1.7.3
• rke2-bootstrap v0.6.1 (upgraded from v0.5.0)
• rke2-control-plane v0.6.1 (upgraded from v0.5.0)

Related: https://github.com/rancher/fleet/issues/2887

Thanks in advance.

[ferhatguneri]

I see the same error in Job apply-system-agent-upgrader-on-clusername-wrk-cjmrg-lvk-93166

(combined from similar events): Error creating: pods "apply-system-agent-upgrader-on-clusername-wrk-cjmrg-lvk-cm7d4" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true, hostIPC=true), privileged (container "upgrade" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "upgrade" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "upgrade" must set securityContext.capabilities.drop=["ALL"]; container "upgrade" must not include "CAP_SYS_BOOT" in securityContext.capabilities.add), restricted volume types (volume "host-root" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "upgrade" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "upgrade" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

[ferhatguneri]

Problem Solved by injecting Excluded namespaces to kube-apiserver-args. Of course, you need to find a way to put these values to cluster spec during the cluster creation. This is just an example to show it works.

cat <<EOF > /var/lib/rancher/rke2/etc/config-files/rancher-psa.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: PodSecurity
    configuration:
      apiVersion: pod-security.admission.config.k8s.io/v1
      kind: PodSecurityConfiguration
      defaults:
        enforce: "restricted"
        enforce-version: "latest"
        audit: "restricted"
        audit-version: "latest"
        warn: "restricted"
        warn-version: "latest"
      exemptions:
        usernames: []
        runtimeClasses: []
        namespaces: [cattle-alerting,
                     cattle-fleet-local-system,
                     cattle-fleet-system,
                     cattle-global-data,
                     cattle-impersonation-system,
                     cattle-monitoring-system,
                     cattle-prometheus,
                     cattle-resources-system,
                     cattle-system,
                     cattle-ui-plugin-system,
                     cert-manager,
                     cis-operator-system,
                     fleet-default,
                     ingress-nginx,
                     kube-node-lease,
                     kube-public,
                     kube-system,
                     rancher-alerting-drivers]
EOF 

apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
    name: # Define cluster name
spec:
    rkeConfig:
        machineSelectorConfig:
        - config:
          kube-apiserver-arg:
            - admission-control-config-file=/var/lib/rancher/rke2/etc/config-files/rancher-psa.yaml
          profile: cis

[jisnardo]

Thanks @ferhatguneri, now the question is how "find a way to put these values to cluster spec during the cluster creation". Meanwhile a cheating way could be postRKE2Commands while expecting an option.