search

pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.24k stars 499 forks source link

[Documentation] A guide on how to safely remove nodes. #566

Closed kkimdev closed 5 years ago

kkimdev commented 5 years ago

The current operation guide doesn't explain how to safely downscale TIDB. I suspect that it's not as simple as just draining the nodes to remove as it the data migration can take more than the default pod termination grace period(30sec) https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods . But in any case, it will be great to have it documented.

tennix commented 5 years ago

@kkimdev I don't know what you mean by nodes? TiDB cluster pods or just k8s work nodes?

For scaling in TiDB cluster pods, it is as simple as changing the component replicas and run helm upgrade as documented in the operation guide.

As for removing k8s worker nodes, there are some extra steps. We'll add the document soon. But if you are familiar with k8s, then after the component replica count is decreased, removing the k8s node can simply be achieved by kubectl drain node command.

weekface commented 5 years ago

https://github.com/pingcap/docs-cn/pull/1417