search

pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.22k stars 493 forks source link

tikv-scaler need to auto delete old PVCs after scale in. #908

Closed shonge closed 4 years ago

shonge commented 5 years ago

Feature Request

Is your feature request related to a problem? Please describe:

There are two tidb-clusters A and B sharing one k8s cluster. When we scale-in cluster A tikv, the old PVCs aren't deleted. And cluster B can't scale-out because it's short of PVs.

Describe the feature you'd like:

When scaling-in tikv, delete the old PVCs.

Describe alternatives you've considered:

Add an option, let user choice defer deleting PVC policy.

Teachability, Documentation, Adoption, Migration Strategy:

https://github.com/pingcap/tidb-operator/blob/799930d59dd86c5d60f39bd42f5ed46761cf0639/pkg/manager/member/tikv_scaler.go#L44-L184

Yisaer commented 5 years ago

Could you provide the steps to reproduce the problem you meet?

weekface commented 5 years ago

When TiKV scale-in, we don't delete the PVC immediately for data safety. When scale-out again, the tidb operator will delete the PVC automatically.

By default, when a local PV is released, the provisioner recycles it. To prevent data from being recycled automatically, the tidb operator set the reclaim policy of your storage class to Retain to prevent data from being automatically deleted.

After confirming that a PV’s data can be deleted, user should modify its reclaim policy to Delete. And then the local-volume-provisioner will delete the data and PV object automatically.

Just delete the PVC will not release the PV and its data. So user should: (1) delete the PVC and (2) set PV reclaim policy to Delete to release the PV. The tidb operator do not do these acitons automatically now.

shonge commented 5 years ago

Thank you, I had already set PV reclaim policy to Delete. But the problem is the PVCs aren't deleted by automatically and other tidb-clusters can't use these PVs.

weekface commented 5 years ago

All right, add an option to control whether defer deleting the PVC. I am fine with this feature.

weekface commented 5 years ago

You must set the PV reclaim policy to Retain in production environment: https://pingcap.com/docs/v3.0/tidb-in-kubernetes/reference/configuration/local-pv/#data-security

Only change PV reclaim policy to Delete manually after confirming that a PV’s data can be deleted.