The etcd restart too often (compared to kubeadm installation method)

[lance5890]

1. I found the etcd restart too often ,compared to kubeadm installation method
2. Maybe this has something to do with the health check , in the ocp, the health check is conducted by the ceo func,

I will complete this issue when I found more evidence

[lance5890]

@Elbehery @tjungblu

[Elbehery]

Thanks @lance5890 for filling this

Please give us more details whenever possible :)

[lance5890]

1. By the kubeadm installation, the etcd health check is conducted as follows, which use the localhost 2381 metrics port as the health check :

   

2. But in the ocp installation, the etcd health check is conducted by the etcd-health-monitor, which use the WithQuorumRead func to check the etcd health as follows:

https://github.com/openshift/cluster-etcd-operator/blob/7a8db9c7132dd171a3a9fd31504cedffc1ea5af5/pkg/cmd/monitor/monitor.go#L161-L171

3. The ocp use etcd QuorumRead to check the etcd health is very heavy operation, as we all now, the etcd quorum is Linearizable，it is heavily dependent on cluster performance
4. If the ocp cluster use hdd disk(not the ssd), and the health check will fail too often, and in this time , restart the etcd will exacerbate cluster instability

[lance5890]

Maybe we should use the light way to check the etcd health , but not depend on the cluster performance @Elbehery @tjungblu

[Elbehery]

any updates here ?

[lance5890]

any updates here ?

the ocp etcd health check is conducted by too heavy, compared to the metrics way by the kubeadm

[Elbehery]

@lance5890 to make this decision we need some performance proofs imho

cc @tjungblu @hasbro17 @dusk125

[lance5890]

as release 4.12 has no health-monitor container any more, i will close this issue and check