`MutatingWebhook` namespace selector missing, giving it wider scope than intended

[ca-scribner]

Upstream Kubeflow applies a namespace selector in their manifests of:

  namespaceSelector:
    matchLabels:
      app.kubernetes.io/part-of: kubeflow-profile

which restricts the admission-webhook to apply only to namespaces which have been created by kubeflow-profiles for users. Currently, we have this webhook to apply to all pods in all namespaces. I think this configuration is why if we break our admission-webook workload (eg: delete the deployment, but do not delete the mutatingwebhook that uses the deployment), it blocks all pods on the cluster from deploying.

We can adopt upstream's configuration by changing our charm's mutatingwebhookconfiguration from:

"mutatingWebhookConfigurations": [
    {
        "name": "admission-webhook",
        "webhooks": [
            {
                "name": "admission-webhook.kubeflow.org",
                "failurePolicy": "Fail",
                "clientConfig": {
                    "caBundle": ca_bundle,
                    "service": {
                        "name": hookenv.service_name(),
                        "namespace": model,
                        "path": "/apply-poddefault",
                        "port": 4443,
                    },
                },
                "objectSelector": {
                    "matchExpressions": [
                        {
                            "key": "juju-app",
                            "operator": "NotIn",
                            "values": ["admission-webhook"],
                        },
                        {
                            "key": "app.kubernetes.io/name",
                            "operator": "NotIn",
                            "values": ["admission-webhook"],
                        },
                        {
                            "key": "juju-operator",
                            "operator": "NotIn",
                            "values": ["admission-webhook"],
                        },
                        {
                            "key": "operator.juju.is/name",
                            "operator": "NotIn",
                            "values": ["admission-webhook"],
                        },
                    ]
                },
                "rules": [
                    {
                        "apiGroups": [""],
                        "apiVersions": ["v1"],
                        "operations": ["CREATE"],
                        "resources": ["pods"],
                    }
                ],
            },
        ],
    }
],

to:

"mutatingWebhookConfigurations": [
    {
        "name": "admission-webhook",
        "webhooks": [
            {
                "name": "admission-webhook.kubeflow.org",
                "failurePolicy": "Fail",
                "clientConfig": {
                    "caBundle": ca_bundle,
                    "service": {
                        "name": hookenv.service_name(),
                        "namespace": model,
                        "path": "/apply-poddefault",
                        "port": 4443,
                    },
                },
                "namespaceSelector": {
                    "matchLabels": {
                        "app.kubernetes.io/part-of": "kubeflow-profile",
                    },
                },
                "rules": [
                    {
                        "apiGroups": [""],
                        "apiVersions": ["v1"],
                        "operations": ["CREATE"],
                        "resources": ["pods"],
                    }
                ],
            },
        ],
    }
],

(remove objectSelector, add namespaceSelector). Strictly speaking, the objectSelector could stay, but I believe it is unnecessary with the namespaceSelector added.

To test this:

• juju deploy admission-webhook
• create a namespace (note that if the selectors are changed to a namespace selector as described above, we must have the correct label in the namespace metadata)

• apiVersion: v1
  kind: Namespace
  metadata:
  name: user
  labels:
  app.kubernetes.io/part-of: kubeflow-profile

• create a PodDefault, for example (the content of this poddefault doesn't matter - this was taken from our kfp charm):

  apiVersion: kubeflow.org/v1alpha1
  kind: PodDefault
  metadata:
  name: access-ml-pipeline
  namespace: user
  spec:
  desc: Allow access to Kubeflow Pipelines
  selector:
  matchLabels:
    access-ml-pipeline: "true"
  volumes:
  - name: volume-kf-pipeline-token
    projected:
      sources:
        - serviceAccountToken:
            path: token
            expirationSeconds: 7200
            audience: pipelines.kubeflow.org      
  volumeMounts:
  - mountPath: /var/run/secrets/kubeflow/pipelines
    name: volume-kf-pipeline-token
    readOnly: true
  env:
  - name: KF_PIPELINES_SA_TOKEN_PATH
    value: /var/run/secrets/kubeflow/pipelines/token

• create a pod with the above cited label of access-ml-pipeline: "true":

  apiVersion: v1
  kind: Pod
  metadata:
  labels:
  access-ml-pipeline: "true"
  name: testpod
  namespace: user
  spec:
  containers:
  - args:
  - "while true; do sleep 3600; done"
  command: ["/bin/bash", "-c", "--"]
  image: ubuntu:latest
  imagePullPolicy: Always
  name: ubuntu

exec a shell in the above pod (kubectl exec -it testpod -- bash) and confirm that /var/run/secrets/kubeflow/pipelines/token exists

Something like the procedure above should be added as an integration test so we confirm PodDefaults actually work correctly

[ca-scribner]

An interesting additional test could be creating a pod in an unlabeled namespace (eg: one that shouldn't be in-scope for the mutating webhook) to make sure the poddefault is not applied