Setting container securityContext has no effect

l0rd commented 4 months ago

Describe the bug

Workspaces containers securityContext can be set in CheCluster, in DevWorkspaceOperatorConfig and in a Devfile as containerOverrides attributes. In particular I am trying to unset it (securityContext: {}). But none of these methods has an effect: the workspace container and initContainer security Context are always set to:

          securityContext:
            capabilities:
              add:
                - SETGID
                - SETUID
              drop:
                - ALL
            runAsNonRoot: true
            readOnlyRootFilesystem: false
            allowPrivilegeEscalation: true

Che version

7.83@latest

Steps to reproduce

Edit CheCluster and set devEnvironments.security.containerSecurityContext to {} or Edit DWOC and set config.workspace.containerSecurityContext to {} or Add in a Devfile the attribute container-overrides: {"securityContext":{}}

Expected behavior

The workspace Pod container security context should be set to the value specified.

Runtime

OpenShift

Screenshots

No response

Installation method

OperatorHub

Environment

Amazon

Eclipse Che Logs

No response

Additional context

No response

ibuziuk commented 3 months ago

@AObuchow could you please take a look?

AObuchow commented 3 months ago

@l0rd I imagine the Che Cluster CR's devEnvironments.DisableContainerBuildCapabilities was set to true when you experienced this behavior?

The documentation for devEnvironments.security.containerSecurityContext: "Requires devEnvironments.disableContainerBuildCapabilities to be set to true in order to take effect.".

Some follow-up thoughts:

Is the current behavior acceptable (i.e. admins need to set devEnvironments.disableContainerBuildCapabilities: true in order to configure the workspaces container security context)?
Should we consider allowing devEnvironments.security.containerSecurityContext to override devEnvironments.disableContainerBuildCapabilities? In other words, the default container build security context would not get applied if devEnvironments.security.containerSecurityContext is configured. Setting the container security context is considered to be an advanced configuration option where we trust the admin knows what they are doing (and that they know they may break their workspaces, and prevent container build capabilities).
Maybe we should enhance the webhook responsible for the Che Cluster CR to prevent users from setting devEnvironments.security.containerSecurityContext if devEnvironments.disableContainerBuildCapabilities is set to false. This would require the most work to implement but would be the most effective solution IMO as it makes the conflict between these two fields explicit to the admin (rather than relying on them to read the documentation).

l0rd commented 3 months ago

@AObuchow I am not able to override the security context even if I set devEnvironments.DisableContainerBuildCapabilities: true. My final goal is to be able to run privileged containers (in a kata-containers context) but I wasn't able to configure Dev Spaces (or DWO) to override the security context.

AObuchow commented 3 months ago

@l0rd Thank you for the follow-up. This must be a bug then. I'll investigate to figure out what's going on here.

AObuchow commented 3 months ago

I've been unable to reproduce this issue on Che 7.84@latest & OpenShift 4.14 so far. Maybe I'm missing a crucial step here?

I've tried:

1. Setting `devEnvironments.DisableContainerBuildCapabilities: true` and configuring `devEnvironments.security.containerSecurityContext` to the following:

    security:
      containerSecurityContext:
        capabilities:
          add:
            - SYS_TIME
            - CHOWN
          drop:
            - KILL

I see that this is reflected in the Che-owned DWOC:

apiVersion: controller.devfile.io/v1alpha1
config:
  workspace:
    containerSecurityContext:
      capabilities:
        add:
          - SYS_TIME
          - CHOWN
        drop:
          - KILL
    deploymentStrategy: Recreate
    persistUserHome:
      enabled: true
    progressTimeout: 300s
    serviceAccount:
      disableCreation: false
kind: DevWorkspaceOperatorConfig
metadata:
(...)
  name: devworkspace-config
  namespace: openshift-operators

And workspace creation fails due to no SCC being available that grants the required privileges, as expected:

Error creating DevWorkspace deployment: Detected unrecoverable deployment condition: FailedCreate pods "workspace4bbd85240e75499b-78d469f4fb-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: "CHOWN": capability may not be added, provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: "SYS_TIME": capability may not be added, provider restricted-v2: .containers[0].capabilities.add: Invalid value: "CHOWN": capability may not be added, provider restricted-v2: .containers[0].capabilities.add: Invalid value: "SYS_TIME": capability may not be added, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]

2. Disabling my custom container security context by setting `devEnvironments.DisableContainerBuildCapabilities: true`.

The Che-owned DWOC is updated as expected:

apiVersion: controller.devfile.io/v1alpha1
config:
  workspace:
    containerSecurityContext:
      allowPrivilegeEscalation: true
      capabilities:
        add:
          - SETGID
          - SETUID
    deploymentStrategy: Recreate
    persistUserHome:
      enabled: true
    progressTimeout: 300s
    serviceAccount:
      disableCreation: false
kind: DevWorkspaceOperatorConfig
(...)

3. Unsetting any container security context with containerBuilds disabled, like in the original issue description:

  devEnvironments:
    startTimeoutSeconds: 300
    security: {}
    secondsOfRunBeforeIdling: -1
    maxNumberOfWorkspacesPerUser: -1
    containerBuildConfiguration:
      openShiftSecurityContextConstraint: container-build
    disableContainerBuildCapabilities: true
    defaultNamespace:
      autoProvision: true
      template: <username>-che
    secondsOfInactivityBeforeIdling: 1800
    storage:
      pvcStrategy: per-user
    persistUserHome:
      enabled: true

The resulting Che-owned DWOC:

apiVersion: controller.devfile.io/v1alpha1
config:
  workspace:
    deploymentStrategy: Recreate
    persistUserHome:
      enabled: true
    progressTimeout: 300s
    serviceAccount:
      disableCreation: false
kind: DevWorkspaceOperatorConfig

Note: I did notice that I had to restart my workspace that was failing from my previously-configured custom security context (that didn't have a valid associated SCC) 2-3 times in order to get the security context to be un-set.

The actual workspace pod ended up getting the restricted-v2 OpenShift SCC, and had the following (expected) security context:

      securityContext:
        capabilities:
          drop:
            - ALL
        runAsUser: 1000690000
        runAsNonRoot: true
        readOnlyRootFilesystem: false
        allowPrivilegeEscalation: false

AObuchow commented 3 months ago

@l0rd if you're able to still reproduce this issue, could you please share:

The version of OpenShift you're deploying Che to
Your Che Cluster CR when the bug state occurs
Any previous reproducer steps that might be relevant. I tried reproducing your issue on a fresh Che install.
Che-Operator logs

The Che-Operator logs seem less helpful than I had hoped: updating the security context field only results in the following being logged:

2024-04-09T18:36:37Z INFO sync Object updated {"namespace": "openshift-operators", "kind": "v1alpha1.DevWorkspaceOperatorConfig", "name": "devworkspace-config"}
time="2024-04-09T18:36:38Z" level=info msg="Successfully reconciled."

Though maybe there's some errors being logged that might be useful for investigation.

l0rd commented 2 weeks ago

Closing as I am not able to reproduce. Thanks @AObuchow for your help.

eclipse-che / che