[RKE2 snapshot] Cluster health check fail error but cluster is able to restore and become active - Githubissues

rancher / dashboard

The Rancher UI

https://rancher.com

Apache License 2.0

460 stars 260 forks source link

[RKE2 snapshot] Cluster health check fail error but cluster is able to restore and become active #5535

Open timhaneunsoo opened 2 years ago

timhaneunsoo commented 2 years ago

Rancher Server Setup

Rancher version: v2.6-head d604b7e
Installation option (Docker install/Helm Chart):
- If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE2
Proxy/Cert Details:

Information about the Cluster

Kubernetes version: v1.21.10+rke2r2
Cluster Type (Local/Downstream):
- If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): AWS EC2
  When restoring a snapshot, the process is successful and the cluster does seem to be restored to the snapshot version, but there is an alarming error while the snapshot restore is in process. There is a Cluster health checks failed error that is there for quite a bit during restoration. This error may make customers worry and think their cluster or Rancher is broken
  
  To Reproduce
  1. deploy a downstream cluster with dedicated nodes per role 2. take a snapshot after the cluster gets to active state 3. deploy a workload, with a nodeport 4. take another snapshot after the workloads are healthy 5. restore the cluster using the 1st snapshot 6. check status of cluster for errors **Result** There is a `Cluster health checks failed` error for a noticeable period **Expected Result**
  The cluster should restore successfully with no errors
  
  Screenshots

paynejacob commented 2 years ago

the solution for this is to suppress health check errors during a snapshot restore. The errors are correct because the cluster is stopped while the restore takes place.