br: clean volumes when restore volume failed

[WangLe1321]

What problem does this PR solve?

Add cleaning volumes function when restore volume failed. Then we can avoid volume leak

Closes #5638

What is changed and how does it work?

Code changes

• [x] Has Go code change
• [ ] Has CI related scripts change

Tests

• [ ] Unit test
• [ ] E2E test
• [x] Manual test
• [ ] No code

manual test steps:

1. create a cluster
2. create a volume backup
3. create a restore cluster
4. create a volume restore using the volume backup in step 2. in this step, restore volume is successful and the tikv pods are started
5. edit the restore CR, modify its status from VolumeComplete to Failed to mock restore volume failed scenario
6. edit the tc CR, remove the annotation tidb.pingcap.com/tikv-volumes-ready to block tikv creation, then delete the tikv statefulset to detach the EBS volumes
7. wait for the volumes detached and deleted by tidb-operator

Side effects

• [ ] Breaking backward compatibility
• [ ] Other side effects:

Related changes

• [ ] Need to cherry-pick to the release branch
• [ ] Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

[WangLe1321]

/run-pull-e2e-kind-br

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 83 lines in your changes are missing coverage. Please review.

Project coverage is 47.81%. Comparing base (3897095) to head (bf2e796). Report is 3 commits behind head on master.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #5634 +/- ## =========================================== - Coverage 61.46% 47.81% -13.65% =========================================== Files 235 219 -16 Lines 30397 30307 -90 =========================================== - Hits 18683 14492 -4191 - Misses 9840 14095 +4255 + Partials 1874 1720 -154 ``` | [Flag](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5634/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | | |---|---|---| | [e2e](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5634/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `47.81% <0.00%> (?)` | | | [unittest](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5634/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `?` | |

[BornChanger]

/retest

[BornChanger]

/retest

[BornChanger]

/run-pull-e2e-kind-basic

[ti-chi-bot[bot]]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BornChanger, csuzhangxc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/pingcap/tidb-operator/blob/master/OWNERS)~~ [BornChanger,csuzhangxc] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[ti-chi-bot[bot]]

[LGTM Timeline notifier]

Timeline:

• 2024-05-04 02:50:36.183869862 +0000 UTC m=+671189.941005436: :ballot_box_with_check: agreed by BornChanger.
• 2024-05-04 02:51:05.269796707 +0000 UTC m=+671219.026932276: :heavy_multiplication_x::repeat: reset by ti-chi-bot[bot].
• 2024-05-06 03:40:45.043388677 +0000 UTC m=+846998.800524249: :ballot_box_with_check: agreed by csuzhangxc.

[csuzhangxc]

/cherry-pick release-1.5

[ti-chi-bot]

@csuzhangxc: new pull request created to branch release-1.5: #5639.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5634#issuecomment-2095144615): >/cherry-pick release-1.5 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.