search

kubernetes-sigs / kubespray

Deploy a Production Ready Kubernetes Cluster
Apache License 2.0
16.14k stars 6.47k forks source link

Adding new node to etcd in existing K8s cluster via Kubespray #11592

Open Chrisys93 opened 1 month ago

Chrisys93 commented 1 month ago 
          > @elgamal2020 I found that if you change the addresses and rerun the playbook, etcd will still try to connect to the previous members. The error message is misleading as the correct address is shown, but etcd may be trying old peers. Running `journalctl -e -u etcd` will show if it's still trying to connect to old member peers. I solved it by deleting the `/var/lib/etcd/member` etcd data directory and killing the `etcd1` docker container on each etcd host. Docker will restart the container and rebuild the data directory. The old members should be gone and you can rerun the playbook.

This works, but sadly I had to reset the whole cluster to re-add a node. Is there any way of updating etcd without having to reset the cluster? Does anyone know? Now I have to set everything - monitoring, orchestration, storage, automation...back up...

Originally posted by @Chrisys93 in https://github.com/kubernetes-sigs/kubespray/issues/6875#issuecomment-2379667974

kokyhm commented 4 weeks ago

Please check: Adding an etcd node Would this work for you?

Chrisys93 commented 3 weeks ago

I tried initially, but did not limit it only to the node to be added. This approach is still pending test for me. I have only managed to reset the cluster and reconnect all nodes, sadly, for now...