Provide a means to add labels and annotations on `stunnerd` pods

[rg0now]

Currently there is no configurable way to add user-defined labels and annotations to the stunnerd pods launched by STUNner to run the data plane for a Gateway. This would be useful, for instance, to enable/disable automatic sidecar proxy injection (e.g., for linkerd) or specify the sidecar proxy parameters.

One possible workflow to support this feature would be to create the Gateway from a separate Dataplane CRD and then simply copy all labels and annotations from the Dataplane into the pods created for the Gateway. If you want different labels/annotations for different Gateways, launch them from a different Dataplane via the GatewayConfig.

This issue is to initiate the discussion on this feature and track implementation.

[rg0now]

Eventually the decision was made to not copy labels/annotations from the Dataplane to the stunnerd Deployments. First, the same Dataplane is often shared across multiple Gateways and it seems wrong to copy the same labels/annotations to every Deployment spawned from a Dataplane. Instead, copying from the Gateway lets users customize stunnerd Deployment labels/annotations on a per-Gateway basis. Second, changing this may break existing deployments that rely on this behavior.

Instead, we cleaned up the way labels and annotations are copied between Gateways and Deployments, see 9a6729d.

Rules for the labels/annotations on the stunnerd Deployment:

• mandatory labels/annotations are enforced (see details in internal/renderer/dataplane_util.go)
• labels/annotations copied verbatim from the parent Gateway unless in conflict with the mandatory labels/annotations
• labels/annotations manually added to the deployment are retained unless in conflict with the above

Rules for the labels/annotations on the pod template for the stunnerd Deployment:

• pod template labels are fixed and there is no way to manually add labels to the pods: this is needed to make sure that the operator reliably finds the pods it manages
• mandatory pod template annotations are enforced (see details in internal/renderer/dataplane_util.go)
• pod template annotations copied verbatim from the parent Gateway unless in conflict with the mandatory annotations
• pod template annotations are copied from the manually added pod template annotations unless in conflict with the above

[rg0now]

After repeated requests from users dissatisfied with the current state-of-the-art that there is no way to customize stunnerd pod labels/annotations, we again changed the corresponding behavior of the operator.

Previously we have ruled out copying labels/annotations from the Dataplane on the basis that "the same Dataplane is often shared across multiple Gateways and it seems wrong to copy the same labels/annotations to every Deployment spawned from a Dataplane." In addition, we found that Dataplane resources often get, from our perspective completely random labels/annotations in operation (e.g., Helm and ArgoCD all add their specific annotations) and we didn't want these random events to induce a full dataplane restart (which would happen if we were to copy these to the pod template).

Eventually, we decided to still adopt this (suboptimal) approach with a little twist: instead of copying the labels/annotations from the Dataplane we rather added to fields two the Dataplane spec and made custom pod labels/annotations explicit. This rules out random pod restarts. At the same time, adding Gateway-specific labels/annotations is still difficult (you have to create a separate Dataplane per each Gateway), but it still seems like a tolerable compromise.

The new rules for the stunnerd pod labels/annotations are as follows:

• Pod labels can be set from the labels field in the Dataplane spec. Mandatory labels override custom pod labels set via the Dataplane spec on conflict, and labels manually added to stunnerd pods are removed on the next update.
• Pod annotations can be set from the annotations field in the Dataplane spec. On conflict mandatory annotations override custom pod annotations set via the Dataplane spec, which in turn override annotations manually added to the pod. Otherwise, manually added pod annotations are preserved.