Configure the CAPA default apps to use the new registry

[alex-dabija]

Task

Configure the CAPA default apps to use the new registry.

Links

• Epic;
• State after initial investigation.

[fiunchinho]

These need to be merged, but our e2e tests are currently broken

• aws-pod-identity-webhook
• etcd-kubernetes-resources-count-exporter
• external-dns
• teleport-kube-agent
• vertical-pod-autoscaler-app

Other things that need to be changed

• The catalog configuration need to be updated to use the new registry. https://github.com/giantswarm/management-cluster-bases/pull/81
• We need a new version of cluster-apps-operator that deploys newer version of chart-operator using the new registry.
• New release of cluster-autoscaler-app with the registry change.
• Upgrade observability-bundle to upgrade apps to use new registry.

[fiunchinho]

Currently blocked until e2e tests are working again and PR's upgrading apps can be tested and merged.

[AndiDog]

All images use the new registry now, tested on a new WC with default-apps-aws v0.48.0:

$ kubectl get pods -A -o jsonpath="{.items[*].spec['initContainers', 'containers'][*].image}" | tr -s '[[:space:]]' '\n' | sort | uniq -c | sort -rn
  56 gsoci.azurecr.io/giantswarm/cilium:v1.14.5
  10 gsoci.azurecr.io/giantswarm/cert-exporter:2.8.5
   9 gsoci.azurecr.io/giantswarm/node-exporter:v1.3.1
   9 gsoci.azurecr.io/giantswarm/net-exporter:1.18.2
   9 gsoci.azurecr.io/giantswarm/docker-kubectl:1.25.4
   9 gsoci.azurecr.io/giantswarm/capi-node-labeler-app:0.5.0
   6 gsoci.azurecr.io/giantswarm/livenessprobe:v2.11.0
   6 gsoci.azurecr.io/giantswarm/aws-ebs-csi-driver:v1.21.0
   5 gsoci.azurecr.io/giantswarm/csi-node-driver-registrar:v2.9.0
   5 gsoci.azurecr.io/giantswarm/aws-ebs-csi-volume-limiter:0.1.0
   4 gsoci.azurecr.io/giantswarm/promtail:2.9.2
   3 gsoci.azurecr.io/giantswarm/kyvernopre:v1.11.4
   3 gsoci.azurecr.io/giantswarm/kyverno:v1.11.4
   3 gsoci.azurecr.io/giantswarm/kube-scheduler:v1.25.16
   3 gsoci.azurecr.io/giantswarm/kube-controller-manager:v1.25.16
   3 gsoci.azurecr.io/giantswarm/kube-apiserver:v1.25.16
   3 gsoci.azurecr.io/giantswarm/etcd:3.5.9-0
   3 gsoci.azurecr.io/giantswarm/coredns:1.11.1
   3 gsoci.azurecr.io/giantswarm/aws-cloud-controller-manager:v1.25.14
   3 gsoci.azurecr.io/giantswarm/amazon-eks-pod-identity-webhook-gs:v0.2.0
   2 gsoci.azurecr.io/giantswarm/vpa-admission-controller:1.0.0
   2 gsoci.azurecr.io/giantswarm/teleport-distroless:14.1.3
   2 gsoci.azurecr.io/giantswarm/prometheus-config-reloader:v0.71.2
   2 gsoci.azurecr.io/giantswarm/metrics-server:v0.6.4
   2 gsoci.azurecr.io/giantswarm/kubectl:1.28.5
   2 gsoci.azurecr.io/giantswarm/grafana-agent:v0.37.2
   2 gsoci.azurecr.io/giantswarm/configmap-reload:v0.8.0
   2 gsoci.azurecr.io/giantswarm/cilium-operator-aws:v1.14.5
   2 gsoci.azurecr.io/giantswarm/chart-operator:3.1.3
   2 gsoci.azurecr.io/giantswarm/cert-manager-webhook:v1.13.3
   1 gsoci.azurecr.io/giantswarm/vpa-updater:1.0.0
   1 gsoci.azurecr.io/giantswarm/vpa-recommender:1.0.0
   1 gsoci.azurecr.io/giantswarm/reports-controller:v1.11.4
   1 gsoci.azurecr.io/giantswarm/prometheus:v2.47.1
   1 gsoci.azurecr.io/giantswarm/prometheus-operator:v0.71.2
   1 gsoci.azurecr.io/giantswarm/policy-reporter:2.17.5
   1 gsoci.azurecr.io/giantswarm/policy-reporter-ui:1.9.2
   1 gsoci.azurecr.io/giantswarm/policy-reporter-kyverno-plugin:1.6.2
   1 gsoci.azurecr.io/giantswarm/kyverno-policy-operator:0.0.7
   1 gsoci.azurecr.io/giantswarm/kube-state-metrics:v2.10.1
   1 gsoci.azurecr.io/giantswarm/hubble-ui:v0.12.1
   1 gsoci.azurecr.io/giantswarm/hubble-ui-backend:v0.12.1
   1 gsoci.azurecr.io/giantswarm/hubble-relay:v1.14.5
   1 gsoci.azurecr.io/giantswarm/external-dns:v0.11.0
   1 gsoci.azurecr.io/giantswarm/etcd-kubernetes-resources-count-exporter:1.9.0
   1 gsoci.azurecr.io/giantswarm/csi-resizer:v1.8.1
   1 gsoci.azurecr.io/giantswarm/csi-provisioner:v3.6.1
   1 gsoci.azurecr.io/giantswarm/csi-attacher:v4.4.1
   1 gsoci.azurecr.io/giantswarm/cluster-autoscaler:v1.27.3
   1 gsoci.azurecr.io/giantswarm/cert-manager-controller:v1.13.3
   1 gsoci.azurecr.io/giantswarm/cert-manager-cainjector:v1.13.3
   1 gsoci.azurecr.io/giantswarm/background-controller:v1.11.4
   1 gsoci.azurecr.io/giantswarm/aws-pod-identity-restarter:1.14.1