Mutating webhook should handle only relevant requests

[shreyas-s-rao]

What would you like to be added:

I would like Kupid's mutating webhook to only handle the requests that are relevant to it by using an ObjectSelector in the webhook configuration. The object selector can be set based on the PSPs and CPSPs that Kupid uses to mutate these resources.

Why is this needed:

Today Kupid receives every request in the cluster, while it only wishes to mutate specific resources (like etcd statefulset) based on resource labels. This allows for low resource consumption by Kupid (by avoiding irrelevant requests to it) and reduces log load by getting rid of unnecessary Handling request... logs.

[shreyas-s-rao]

Quoted from https://github.com/gardener/kupid/pull/54#discussion_r1151818191 and https://github.com/gardener/kupid/pull/54#discussion_r1153199430

@unmarshall raised a valid point about setting object selectors in webhooks:

Use the object selector only if the webhook is opt-in, because end users may skip the admission webhook by setting the labels.

For the gardener use case, we are the end-users of Kupid as well. So we control what labels we set on the pod-groups (statefulsets, deployments, etc) that Kupid is to mutate.

This can be achieved using the helm charts. We can read the {{ .Values.policies.*.podLabels }} section from each policy and pass that as a set to Kupid as a command-line argument. Kupid can simply set these on the webhooks that it creates.

Now, external users of Kupid can choose to leave this empty, in which case Kupid will not add any object selector to the webhooks, which means all resources will be considered (no change in behavior).

[shreyas-s-rao]

/assign

[shreyas-s-rao]

/unassign