Isolation improvements: dynamically create ValidatingWebhookConfiguration

[dougbtv]

Currently we have a situation where we're creating the ValidatingWebhookConfiguration up front, and we're setting it up to listen for all pod creation events on all namespaces.

See: https://github.com/K8sNetworkPlumbingWG/net-attach-def-admission-controller/blob/44f8ae8cbe2d87884b91d028e0ffca1e8ab2f094/deployments/webhook.yaml#L2-L18

This leaves for a situation where there's a failure of the net-attach-def-admission-controller (for example, the pod gets killed and for some reason it doesn't come back -- say, a misconfiguration), and we deny any pods from being created cluster-wide.

In order to mitigate this failure, I propose that we dynamically create the ValidatingWebhookConfiguration for the isolation feature (this is the feature that listens to pod creation events) -- and we limit the ValidatingWebhookConfiguration to listen on specific namespace(s) -- only namespaces with NetworkAttachmentDefinitions defined within them. And we create one for each namespace, or modify the namespaces under which we gate pod creation.

Psuedocode (I have not trialed nor validated this, just stubbing in the "namespaces" key) in yaml for the filtering to specific namespaces (specifically note the last line):

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: net-attach-def-admission-controller-isolating-config
webhooks:
  - name: net-attach-def-admission-controller-isolating-config.k8s.io
    clientConfig:
      service:
        name: net-attach-def-admission-controller-service
        namespace: ${NAMESPACE}
        path: "/isolate"
      caBundle: ${CA_BUNDLE}
    rules:
      - operations: [ "CREATE" ]
        apiGroups: ["apps", ""]
        apiVersions: ["v1"]
        resources: ["pods"]
        namespaces: ["foo","bar","quux"]

I envision that there will be two processes in order to dynamically create these ValidatingWebhookConfigurations.

1. We must have an initialization process that happens when the net-attach-def-admission-controller is first launched which looks at the ValidatingWebhookConfiguration, then looks at all NetworkAttachmentDefinitions and determines which namespaces have net-attach-defs -- it then reconciles this ValidatingWebhookConfiguration with the namespaces containing net-attach-defs.
2. When we get the creation of NetworkAttachmentDefinitions -- we also run this same reconcilation process, and add any new namespaces to the list (as shown in psuedocode above).

Upside: This greatly mitigates a failure of the net-attach-def-admission-controller at scale for namespaces (or entire deployments) that do not have defined net-attach-defs.

Downside: The namespace isolation feature in Multus must still be used. As this alone will not be enough for security purposes.

@s1061123 -- btw, I'll pick this up when I'm back from PTO!

[s1061123]

@dougbtv sure, got it! looking forward to see that!