search

konstellation-io / kre

Konstellation Runtime Engine
https://www.konstellation.io
16 stars 2 forks source link

KRE v8.2.0 will not deploy on CIS enabled k8s clusters (runAsNonRoot) #724

Open haunted-pancakes opened 1 year ago

haunted-pancakes commented 1 year ago

Environmental Info: RKE2 Version: rke2 version v1.24.12+rke2r1

Kubernetes version:

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.12+rke2r1", GitCommit:"ef70d260f3d036fc22b30538576bbf6b36329995", GitTreeState:"clean", BuildDate:"2023-03-17T18:01:17Z", GoVersion:"go1.19.7 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}

KRE Version: v8.2.0

Cluster Configuration: 1 Controller, 2 workers with CIS 1.6 profile enabled.

https://docs.rke2.io/security/hardening_guide

Describe the bug: After running helm install, pods will not schedule due to global psp configuration which adds runAsNonRoot to the spec (see CIS 1.6 Self-Assessment Guide), or admission-control-config-file (CIS 1.23 Self-Assessment Guide).

Steps To Reproduce:

Expected behavior: Pods will schedule on CIS hardened platform.

Actual behavior: Pods will not schedule, helm install fails after timeout.

Additional context / logs: Most events look like: container has runAsNonRoot and image will run as root (pod: "kre-admin-api-7d499b76bf-d5gvs_kre(ae38ef4e-1841-4aa6-8187-40874b388941)", container: create-admin-api-files-directory)

or

container has runAsNonRoot and image has non-numeric user (kre), cannot verify user is non-root (pod: "kre-k8s-manager-865bdb4bf5-nc8jb_kre(fe9b5463-bac2-449f-8eb3-da80bcdc453e)", container: kre-k8s-manager)

All Deployment, Pod yaml attached as zip. kre.zip

Here's an abbreviated example of a pod spec:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    checksum/config: 29aa9a7b3e6d830e619edc73fe0e6f6831e60b6b397406656f97b156f6d719cc
    cni.projectcalico.org/containerID: f9399fe68208532ca8aa7d5d19973c952cca25619f37ece4e6bfb8921855440d
    cni.projectcalico.org/podIP: 10.42.27.110/32
    cni.projectcalico.org/podIPs: 10.42.27.110/32
    kubernetes.io/psp: global-restricted-psp
  creationTimestamp: '2023-05-15T20:42:54Z'
  generateName: kre-admin-ui-7fbf747487-
  labels:
    app.kubernetes.io/instance: kre
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kre-admin-ui
    app.kubernetes.io/version: v8.2.0
    helm.sh/chart: kre
    pod-template-hash: 7fbf747487
    type: admin
...
  name: kre-admin-ui-7fbf747487-87b98
  namespace: kre
  ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: kre-admin-ui-7fbf747487
      uid: 37715295-06d0-4fb0-869f-cb11183dc593
  resourceVersion: '300760'
  uid: 196d4922-52fa-4007-bfae-103a0100db40
spec:
  containers:
    - image: konstellation/kre-admin-ui:v8.2.0
      imagePullPolicy: IfNotPresent
      name: kre-admin-ui
      ports:
        - containerPort: 8080
          name: web
          protocol: TCP
      resources: {}
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /usr/share/nginx/html/config/config.json
          name: config-volume
          subPath: config.json
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-64kk7
          readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: rke2-node5
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1
    supplementalGroups:
      - 1
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
  volumes:
    - configMap:
        defaultMode: 420
        name: kre-admin-ui
      name: config-volume
    - name: kube-api-access-64kk7
      projected:
        defaultMode: 420
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              items:
                - key: ca.crt
                  path: ca.crt
              name: kube-root-ca.crt
          - downwardAPI:
              items:
                - fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
                  path: namespace
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: '2023-05-15T20:42:55Z'
      status: 'True'
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: '2023-05-15T20:42:55Z'
      message: 'containers with unready status: [kre-admin-ui]'
      reason: ContainersNotReady
      status: 'False'
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: '2023-05-15T20:42:55Z'
      message: 'containers with unready status: [kre-admin-ui]'
      reason: ContainersNotReady
      status: 'False'
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: '2023-05-15T20:42:54Z'
      status: 'True'
      type: PodScheduled
  containerStatuses:
    - image: konstellation/kre-admin-ui:v8.2.0
      imageID: ''
      lastState: {}
      name: kre-admin-ui
      ready: false
      restartCount: 0
      started: false
      state:
        waiting:
          message: >-
            container has runAsNonRoot and image has non-numeric user (nginx),
            cannot verify user is non-root (pod:
            "kre-admin-ui-7fbf747487-87b98_kre(196d4922-52fa-4007-bfae-103a0100db40)",
            container: kre-admin-ui)
          reason: CreateContainerConfigError
  hostIP: 192.168.174.15
  phase: Pending
  podIP: 10.42.27.110
  podIPs:
    - ip: 10.42.27.110
  qosClass: BestEffort
  startTime: '2023-05-15T20:42:55Z'

Plus here is a copy of the global psp:

Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
Name:  global-restricted-psp

Settings:
  Allow Privileged:                        false
  Allow Privilege Escalation:              false
  Default Add Capabilities:                <none>
  Required Drop Capabilities:              ALL
  Allowed Capabilities:                    <none>
  Allowed Volume Types:                    configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
  Allow Host Network:                      false
  Allow Host Ports:                        <none>
  Allow Host PID:                          false
  Allow Host IPC:                          false
  Read Only Root Filesystem:               false
  SELinux Context Strategy: RunAsAny
    User:                                  <none>
    Role:                                  <none>
    Type:                                  <none>
    Level:                                 <none>
  Run As User Strategy: MustRunAsNonRoot
    Ranges:                                <none>
  FSGroup Strategy: MustRunAs
    Ranges:                                1-65535
  Supplemental Groups Strategy: MustRunAs
    Ranges:

Let me know if you need anything else!