[t8s-cluster] use cilium's kube-proxy replacement

[cwrau]

This needs the APIServer URL, see https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/

We could use https://gitlab.teuto.net/4teuto/ops/k8s/t8s-engine/-/issues/13 for this, which would implement a DNS record for each clusters' API server

After we implement this, we need to make sure the user doesn't switch back to calico, as we don't have a good way to install kube-proxy in that case.

[cwrau]

I've tried this, and chatted with some SCS people on this topic, but it seems like we definitely need the API IP as a cilium parameter to get this working.

This is even mentioned in the docs, I just hoped it was kinda optional 😅

Currently, I see a couple of options;

• create a configmap that reflects the API IP which we can use in the cilium HelmRelease
  • this can come from the capi cluster via
  • a new HelmRelase and syncs via a CronJob I guess
  • the t8s operator
• we could patch the cilium helm chart to allow for initContainers which we can use to get the API IP from the kubeadm-config configmap