retry evict leader when upgrading TiKV if needed

csuzhangxc commented 2 months ago

What problem does this PR solve?

close #5614

In some cases when upgrading TiKV, the evict-leader-scheduler may not be added or missing in PD, but the evictLeaderBeginTime annotation of the TiKV pod is added. As TiDB Operator will not call PD API to add evict-leader-scheduler again, then the upgrade operation is blocked.

In this PR, we check whether the evict-leader-scheduler exists if a timeout (10m) is reached, and try to add evict-leader-again if it's missing.

What is changed and how does it work?

Code changes

[ ] Has Go code change
[ ] Has CI related scripts change

Tests

[ ] Unit test
[ ] E2E test
[x] Manual test
1. hack TiDB Operator code to remove the controller.GetPDClient(u.deps.PDControl, tc).BeginEvictLeader(storeID) call in beginEvictLeader
2. upgrade TiKV
3. check whether the upgrade operation is blocked (wait for evictition to complete), and check the evictLeaderBeginTime annotation existing
4. revert the hack in step.1 and re-deploy TiDB-Operator
5. wait about 10m
6. check evictLeaderBeginTime is updated to the new time
7. check upgrade operation is continue and completed later
[ ] No code

Side effects

[ ] Breaking backward compatibility
[ ] Other side effects:

Related changes

[ ] Need to cherry-pick to the release branch
[ ] Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

csuzhangxc commented 2 months ago

/run-all-tests

codecov-commenter commented 2 months ago

Codecov Report

Attention: Patch coverage is 0% with 12 lines in your changes are missing coverage. Please review.

Project coverage is 24.37%. Comparing base (72bccb5) to head (432a9a2). Report is 2 commits behind head on master.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #5613 +/- ## =========================================== - Coverage 61.49% 24.37% -37.12% =========================================== Files 235 219 -16 Lines 30337 30208 -129 =========================================== - Hits 18655 7364 -11291 - Misses 9813 21818 +12005 + Partials 1869 1026 -843 ``` | [Flag](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5613/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | | |---|---|---| | [e2e](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5613/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `24.37% <0.00%> (?)` | | | [unittest](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5613/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `?` | |

csuzhangxc commented 2 months ago

/run-pull-e2e-kind-across-kubernetes

ti-chi-bot[bot] commented 2 months ago

@ideascf: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5613#pullrequestreview-2000153121): >LGTM Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

ti-chi-bot[bot] commented 2 months ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ideascf Once this PR has been reviewed and has the lgtm label, please ask for approval from csuzhangxc, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/pingcap/tidb-operator/blob/master/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

csuzhangxc commented 2 months ago

/run-pull-e2e-kind-across-kubernetes

csuzhangxc commented 2 months ago

/cherry-pick release-1.5

ti-chi-bot commented 2 months ago

@csuzhangxc: new pull request created to branch release-1.5: #5616.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5613#issuecomment-2056253441): >/cherry-pick release-1.5 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.

pingcap / tidb-operator