CAPA cluster upgrades stability

[alex-dabija]

Story

-As a cluster admin, I want CAPA cluster upgrades to be seamless in order to provide the a good user experience to my customers.

Background

CAPA cluster upgrade have been an ongoing struggle.

Current problems

• https://github.com/giantswarm/roadmap/issues/1720;
• Pods become unauthorized during rolling of control plane nodes;
• Control plane nodes have no health checks configured;
• Machine pools have no health checks configured;
• Proxy environment variables are not injected if Kyverno is down, but the pods still start;

Tasks

• [x] https://github.com/giantswarm/roadmap/issues/1766;
• [x] https://github.com/giantswarm/roadmap/issues/1742;
• [x] https://github.com/giantswarm/roadmap/issues/1753;
• [x] https://github.com/giantswarm/roadmap/issues/1755;
• [x] https://github.com/giantswarm/roadmap/issues/1770;
• [x] https://github.com/giantswarm/roadmap/issues/1756;
• [x] https://github.com/giantswarm/roadmap/issues/1757;
• [x] https://github.com/giantswarm/roadmap/issues/1978;
• [x] https://github.com/giantswarm/roadmap/issues/1825;
• [x] https://github.com/giantswarm/giantswarm/issues/25159;
• [x] https://github.com/giantswarm/roadmap/issues/1827;
• [x] https://github.com/giantswarm/roadmap/issues/2170;
• [x] https://github.com/giantswarm/roadmap/issues/2217;
• [x] https://github.com/giantswarm/roadmap/issues/2111;

[alex-dabija]

Our biggest stability issue during upgrade is the fact that nodes are not rolled automatically: https://github.com/giantswarm/roadmap/issues/2217 . This was intentionally done by upstream. We are working on a fix. See @AndiDog's comment: https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues/4071#issuecomment-1506829277 .