pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.2k stars 490 forks source link

br: clean volumes when restore volume failed (#5634) #5639

Closed ti-chi-bot closed 2 months ago

ti-chi-bot commented 2 months ago

This is an automated cherry-pick of #5634

What problem does this PR solve?

Add cleaning volumes function when restore volume failed. Then we can avoid volume leak

Closes #5638

What is changed and how does it work?

Code changes

Tests

manual test steps:

  1. create a cluster
  2. create a volume backup
  3. create a restore cluster
  4. create a volume restore using the volume backup in step 2. in this step, restore volume is successful and the tikv pods are started
  5. edit the restore CR, modify its status from VolumeComplete to Failed to mock restore volume failed scenario
  6. edit the tc CR, remove the annotation tidb.pingcap.com/tikv-volumes-ready to block tikv creation, then delete the tikv statefulset to detach the EBS volumes
  7. wait for the volumes detached and deleted by tidb-operator

image

image

image

Side effects

Related changes

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

ti-chi-bot[bot] commented 2 months ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please assign tennix for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/pingcap/tidb-operator/blob/release-1.5/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment