Better handle client certificates renewal

Problems

The client certificates generated in EKS-A cluster expire after one year. The cluster stops functioning when certs expire.

Currently, EKS-A renews the client certificates for control plane(CP) whenever the CP machine is rolled-out and regenerated. Therefore, most users won't encounter certs expiry issue as the cluster shall be upgraded within a year. However, if an EKS-A cluster is never upgraded - i.e. the CP or external ETCD machines are never rotated for a whole year, the client certificates will expire.

Solutions

Though we recommend all users to keep their EKS-A clusters up to date, it is still likely some clusters won't be upgraded for long time. To solve this, we need to: 1) provide documentation around the client certs manual renewal steps user can follow to renew both CP certs and external ETCD certs (if any) at any time without cluster going down. 2) provide options to auto-renew all the client certs before they expire. This can be done by rolling out the machines before the certs expiration date, triggering the cert rotation to happen. REF: https://cluster-api.sigs.k8s.io/tasks/certs/auto-rotate-certificates-in-kcp.html

Notice that the CP certs manual/auto renewal process is trivial as it is covered by kubeadm as well as CAPI. But the external ETCD certs management is lacking from etcdadm and etcdadm controllers. We will need to research and potentially implement additional support just for external ETCD certs renewal. Related external ETCD certs issue: https://github.com/aws/eks-anywhere/issues/5400

aws / eks-anywhere

Better handle client certificates renewal #5310