Changing hosts count makes subsequent terraform apply to fail because of etcd

[ivdelchev]

Wonderful guide, thank you!

Quick question - if I have a running cluster of e.g. 3 nodes and change that to 4, then at the etcd step the terraform apply fails because etcd requires a somewhat stateful process to add a new node - ssh to existing node, add new one and on the new one start etcd with the env variables from previous step, more details on etcd cluster reconfiguration. What would be a good way to handle this? Alternatively I'd like to limit etcd to 3 nodes, while there could be many more k8s nodes.

[sandromodarelli]

This guide show how to setup a 3masters cluster. To expand cluster you have only to join nodes, without expanding etcd

[ivdelchev]

Thanks @sandromodarelli for your reply. But this is also my question - how do I do this with the terraform configuration? If I change the hosts count, the process fails because of the way etcd gets bootstrapped and I am not sure how to avoid that step.

[pstadler]

The terraform scripts only deploys a single k8s master node while etcd is distributed across all nodes. I'd suggest you deploy a new cluster instead of increasing the size of it afterwards, although this would be feasible with only minor changes. I'm going to leave this issue open as I'd like to introduce a variable to specify the etcd cluster size at some point.

[ivdelchev]

@pstadler, thanks for the guide, it's been a really enjoyable read!

Resizing a k8s cluster would certainly be useful, sometimes all you need is a bit of extra cpu and adding a node would make it much easier than migrating a whole cluster every time. Also from what I've read etcd should not need more than 3/5/7 nodes and if you spin 100 nodes with this terraform script, then you would have 100 etcd nodes. I am kind of new to terraform, but if I figure it out I will create a PR.