Cannot overwrite existing securityContext in podTemplate

[sampritipanda]

For some web challenges where I don't need a nsjail, I would like to start the container without CAP_SYS_ADMIN. As a result, I modify the securityContext to remove any capabilities using the following:

spec:
  ...
  podTemplate:
    template:
      spec:
        containers:
          - name: challenge
            securityContext:
              capabilities: {}

However, this does not remove the capabilities and the container is still deployed with CAP_SYS_ADMIN. This is because of a bug in kctf-operator code here:

https://github.com/google/kctf/blob/v1/kctf-operator/pkg/controller/challenge/deployment/deployment.go#L73

    if deployment.Spec.Template.Spec.Containers[idx_challenge].SecurityContext.Capabilities == nil {
        deployment.Spec.Template.Spec.Containers[idx_challenge].SecurityContext.Capabilities = &corev1.Capabilities{};
    }

    deployment.Spec.Template.Spec.Containers[idx_challenge].SecurityContext.Capabilities.Add =
        append(deployment.Spec.Template.Spec.Containers[idx_challenge].SecurityContext.Capabilities.Add, "SYS_ADMIN")

In the existing code, regardless of what the Capabilities is set to, it adds the CAP_SYS_ADMIN capability. I understand that this is needed by nsjail, but if I manually specify some capabiltiies, then it should not be overwritten by the kctf-operator.

I'm not sure what the best solution is here, but we should definitely be given an option to not start containers with CAP_SYS_ADMIN.

[sirdarckcat]

@sroettger fixed this on https://github.com/google/kctf/pull/301

[sirdarckcat]

Actually looking at this more closely, the fix we made forces cap_sys_admin on you.

[sirdarckcat]

Idea: add a nsjail: false to challenge.yaml or something to tell the operator to ignore cap_sys_admim

[sroettger]

Idea: add a nsjail: false to challenge.yaml or something to tell the operator to ignore cap_sys_admim

that sounds good to me, but it should probably not say nsjail. Something like: "kctf_default_caps" or so?

Can you drop privileges in your CMD as a workaround?

[sirdarckcat]

Yep! That sounds good. We can bikesched the name a bit. I assume kctf is kindof redundant? What about:

    extra_hardening:
     - apparmor_policy
     - no_capabilities

I dont like the ´no_capabilities´ is a negative, but wdyt of this? @sroettger

[sroettger]

Maybe no_default_capabilities? no_capabilities would imply that we always run without capabilities, but I think it should control that we don't add any capabilities for kctf itself.

For example

capabilities: CAP_NET_ADMIN
extra_hardening:
 - no_capabilities

should still have net_admin at the end.