Kubespray periodic backup etcd data - disaster recovery cluster from etcd backups

[haminhcong]

• [ ] Which Kubernetes cluster data stored in ETCD
• [ ] Monitor etcd data backup process and alert when failed to backup K8S data
• [ ] When recovery, do we need change or keep master IP and Name ?
• [ ] Persistent Volume Backup & Restore ? https://github.com/vmware-tanzu/velero

Data which need to save and backup periodic from K8S Cluster Provisioned By Kubespray

• Inventory File
• Node Password Information
• Etcd Node Data:
  • Etcd backup
  • Etcd certificates

Backup And Monitoring Tools

• https://github.com/gardener/etcd-backup-restore/blob/master/doc/usage/metrics.md

Possible scenario

• Recover etcd cluster from backup data
• Recover first master node
• Join remaining master nodes
• Join remaining worker nodes

Need to search in kubespray slack about this problem and this playbook https://github.com/kubernetes-sigs/kubespray/commit/ac2135e450c731be0d855317cc32d8d8d1e1bf92#diff-3bda861e9e89b5053f64df966fe350fcbbe32a3c5f1b977fdb8f004e28c7d47b

References

• https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/
• https://github.com/kubernetes-sigs/kubespray/pull/4146
• https://github.com/kubernetes-sigs/kubespray/issues/1122
• https://documentation.suse.com/suse-caasp/4.2/html/caasp-admin/_cluster_disaster_recovery.html