search

kopeio / etcd-manager

operator for etcd: moved to https://github.com/kubernetes-sigs/etcdadm
Apache License 2.0
164 stars 45 forks source link

when cluster members are added or removed, we should update the state file #218

Open mmerrill3 opened 5 years ago

mmerrill3 commented 5 years ago

The state file used to initialize the etcd server should reflect when etcd members come and go. Currently, we update /etc/hosts when this information changes, but we do not update the state file under the base data directory for etcd-main or etcd-events. I also propose updating the state.Nodes information for the running etcd service to reflect the changes. This manifests as bad node data being returned by the GetInfo peers service, since the state of the cluster comes from that state file, and it doesn't get updated once its read. If you decide to expand your cluster, and then shrink it (for say, moving to a new subnet), references to the old etcd members will still be seen in the logs when the "master" etcd server calls GetInfo on the other peers.

adammw commented 4 years ago

We're noticing this issue on rolling restarts as well, where we see both old and new IPs listed in the hosts update messages.