Drop capabilities CAP_SYS_ADMIN or all by default

[NissesSenap]

More or less nothing good can come from letting pods have the capabilities CAP_SYS_ADMIN.

CAP_SYS_ADMIN
Perform a range of system administration operations including: [quotactl](https://linux.die.net/man/2/quotactl)(2), [mount](https://linux.die.net/man/2/mount)(2), [umount](https://linux.die.net/man/2/umount)(2), [swapon](https://linux.die.net/man/2/swapon)(2), [swapoff](https://linux.die.net/man/2/swapoff)(2), [sethostname](https://linux.die.net/man/2/sethostname)(2), and [setdomainname](https://linux.die.net/man/2/setdomainname)(2);

Preferably we would drop all by default. This might create a issues on the platform level but at the same time we ignore many of the platform namespaces.

If we are using mutating webhooks that applies drop "all" is it still possible to let the users to apply add? If so that could be an option if our tenants have very special needs.

      containers:
      - name: payment
        image: nginx
        securityContext:
          capabilities:
            drop:
              - all
            add:
              - NET_BIND_SERVICE

DoD:

• Can we use add and a specific capability at the same time as we are using a mutating webhook?
• If so drop all
• If not only drop CAP_SYS_ADMIN to all our containers
• release a new helm chart version

[NissesSenap]

The current constraints don't enforce this. My suggestion is that we disable the default enforce and we instead write our own that includes the config to also drop CAP_SYS_ADMIN. Else we entrust everything to the mutating webhook. What do you think @phillebaba

My initial idea was to DROP all. But If i define a ADD capability I get:

apiVersion: v1
kind: Pod
metadata:
  name: debu22
  namespace: lab
spec:
  containers:
    - command:
        - /bin/sh
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - NET_RAW
          add:
            - NET_BIND_SERVICE
        readOnlyRootFilesystem: true
      resources:
        requests:
          memory: "16Mi"
          cpu: "10m"
        limits:
          memory: "64Mi"
          cpu: "100m"
      image: alpine:latest
      name: container-00
      tty: true

Error from server ([psp-capabilities] container <container-00> has a disallowed capability. Allowed capabilities are [""]): error when creating "debug.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [psp-capabilities] container <container-00> has a disallowed capability. Allowed capabilities are [""]

This is due to k get k8spspcapabilities.constraints.gatekeeper.sh psp-capabilities -o yaml allowedCapabilities=""

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPCapabilities
metadata:
  annotations:
    meta.helm.sh/release-name: gatekeeper-library-constraints
    meta.helm.sh/release-namespace: gatekeeper-system
  creationTimestamp: "2022-01-31T09:22:28Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: psp-capabilities
  resourceVersion: "273515965"
  uid: fe3d5f29-7f21-4489-aeb6-ae72aacd1514
spec:
  enforcementAction: deny
  match:
    kinds:
    - apiGroups:
      - ""
      kinds:
      - Pod
  parameters:
    allowedCapabilities:
    - ""
    requiredDropCapabilities:
    - NET_RAW

We can of course change this. But the question is what should we set it to? Should we DROP all by default but allow users to set whatever they want?

I think if we set drop all we need to capability to ether ignore the rule or allow our users to add capabilities.

[NissesSenap]

So I have read up more on how this helm chart actually works... As it actually says there are very few default values and allot can be overwritten, in our case we do it in: https://github.com/XenitAB/terraform-modules/blob/main/modules/kubernetes/opa-gatekeeper/locals.tf

So If I want to enforce the drop of CAP_SYS_ADMIN or even ALL then I can do that there. If we ever get a need to allow a specific capability we can control this through the same config. Currently we have setup ignore lists for namespace but if we ever need we can always add the possibility of exemptImages to our helmchart.

I'm leaning more towards using the default value of drop all after actually understanding all this.

[NissesSenap]

Verify the Drop the of CAP_SYS_ADMIN https://github.com/XenitAB/terraform-modules/pull/590

[NissesSenap]

This is solved