0.4.0 mutation webhook failure in OpenShift when 'anyuid' SCC is set and when explicit RunAsUser is NOT set

[bejoycalias]

I have a deployment in OpenShift where I want to run some pods with 'anyuid' SCC, but some other pods without 'anyuid' permission. I have added the service account to anyuid SCC, and I explicitly mention the 'RunAsUser' parameter for only certain deployments, and NO 'RunAsUser' for certain other deployment which I want to run as UIDs reserved for my namespace. When I don't explicitly mention 'RunAsUser', it fails with error "Error creating: admission webhook "vault.hashicorp.com" denied the request: error creating new agent sidecar: RunAsUser is nil for Container 0's SecurityContext".

I have tested the below 3 scenarios, first 2 works fine, and 3rd one fails.

1. My service account IS NOT part of 'anyid' SCC, I set the annotation 'vault.hashicorp.com/agent-run-as-same-user: "true"' in the deployment yaml. Injection works perfectly fine, injector and my application pod runs with UIDs reserved for my namespace
2. My service account IS part of 'anyuid' SCC. I set the annotation 'vault.hashicorp.com/agent-run-as-same-user: "true"' in the deployment yaml. My deployment spec has an explicit runAsUser value for the application. Injection works perfectly fine, both the injector and application pod runs as the UID I mentioned in the deployment spec.
3. My service account IS part of 'anyuid' SCC. I set the annotation 'vault.hashicorp.com/agent-run-as-same-user: "true"' in the deployment yaml. My deployment spec DOES NOT have an explicit runAsUser value for the application since I want this app component to run with the default reserved UID for that namespace, but with same service account name. Mutation webhook fails with error "Error creating: admission webhook "vault.hashicorp.com" denied the request: error creating new agent sidecar: RunAsUser is nil for Container 0's SecurityContext".

This is my sample deployment yaml. If I remove the 'RunAsUser' line, it doesn't work. I want to run this specific deployment with the default UIDs of the namespace.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-agent-demo
  labels:
    app: vault-agent-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-demo
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-run-as-same-user: "true"
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-aws.creds: aws/creds/sns
        vault.hashicorp.com/ca-key: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
        vault.hashicorp.com/role: sns-role
        vault.hashicorp.com/secret-volume-path: /tmp/bejoy
      labels:
        app: vault-agent-demo
    spec:
      serviceAccountName: mysa
      containers:
      - name: vault-agent-demo
        image: jweissig/app:0.0.1
        securityContext:
          runAsUser: 100

I am running below versions OpenShift 4.4.4 Injector image: vault-k8s:0.4.0 Vault image: vault:1.4.2

[tvoran]

I want to run this specific deployment with the default UIDs of the namespace.

Instead of run-as-same-user, have you tried vault.hashicorp.com/agent-set-security-context: "false"? That will let openshift apply its own security context for your openshift project. Docs here: https://www.vaultproject.io/docs/platform/k8s/injector/annotations#vault-hashicorp-com-agent-set-security-context

[bejoycalias]

@tvoran I tried setting vault.hashicorp.com/agent-set-security-context: "false" and removed 'run-as-user' annotation, but then the container runs as 'root', which is not right.

[tvoran]

Interesting, so it sounds like under the anyuid SCC, openshift doesn't fill in securityContext.RunAsUser for the containers, even though there is a default UID for the project. We'll have to look into whether or not the injector is able to pull that information from openshift/k8s.

As a workaround, you can add an annotation to your app deployment specifying the userid to run the injected containers as (https://www.vaultproject.io/docs/platform/k8s/injector/annotations#vault-hashicorp-com-agent-run-as-user) though that requires a manual step of looking up the project's UID.