pdms: Choose a suitable pdms to transfer primary when upgrade

HuSharp commented 4 months ago

What problem does this PR solve?

Ref #1235, Ref https://github.com/tikv/pd/pull/8157

What is changed and how does it work?

summary

Let's assume there are three tso nodes scheduling-0, scheduling-1, scheduling-2. tidb-operator will upgrade them in the order 2->0. If scheduling-1 is primary, it is possible that when upgrading scheduling-1, the primary will be transferred to scheduling-0, and then the primary will be transferred again when upgrading scheduling-0.

This pr ensures that when scheduling-1 is upgraded, the primary is transferred to scheduling-2, reducing the number of transfers.

Using API

When I created 3 scheduling pods with 8.3.0 PD version

$ kubectl exec -it basic-pd-0 -n pingcap sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
sh-5.1# curl --location --request GET 'http://127.0.0.1:2379/pd/api/v2/ms/members/scheduling'
[
    {
        "name": "basic-scheduling-0",
        "service-addr": "http://basic-scheduling-0.basic-scheduling-peer.pingcap.svc:2379",
        "version": "v8.3.0",
        "git-hash": "2d9a3b0e5da1a8e50251c4510368e5b3085394c7",
        "deploy-path": "/",
        "start-timestamp": 1723535895
    },
    {
        "name": "basic-scheduling-1",
        "service-addr": "http://basic-scheduling-1.basic-scheduling-peer.pingcap.svc:2379",
        "version": "v8.3.0",
        "git-hash": "2d9a3b0e5da1a8e50251c4510368e5b3085394c7",
        "deploy-path": "/",
        "start-timestamp": 1723535883
    },
    {
        "name": "basic-scheduling-2",
        "service-addr": "http://basic-scheduling-2.basic-scheduling-peer.pingcap.svc:2379",
        "version": "v8.3.0",
        "git-hash": "2d9a3b0e5da1a8e50251c4510368e5b3085394c7",
        "deploy-path": "/",
        "start-timestamp": 1723535831
    }
]

// get current leader which is `scheduling-1`
sh-5.1# curl --location --request GET 'http://127.0.0.1:2379/pd/api/v2/ms/primary/scheduling'
"http://basic-scheduling-1.basic-scheduling-peer.pingcap.svc:2379"

// we need to login `scheduling-1` machine
// and then transfer primary to `scheduling-2`
$ kubectl exec -it basic-scheduling-1 -n pingcap sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
sh-5.1# curl --location --request POST 'http://127.0.0.1:2379/scheduling/api/v1/primary/transfer' \
--header 'Content-Type: application/json' \
--data-raw '{
    "new_primary": "basic-scheduling-2"
}'
"success"

// get current leader which is `scheduling-2`
$ kubectl exec -it basic-pd-0 -n pingcap sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
sh-5.1# curl --location --request GET 'http://127.0.0.1:2379/pd/api/v2/ms/primary/scheduling'
"http://basic-scheduling-2.basic-scheduling-peer.pingcap.svc:2379"

check log

Let's upgrade 3 scheduling, and primary is scheduling-2 now.

// when `scheduling-2` is primary, should transfer to `scheduling-0`
I0813 07:57:08.289779       1 pd_ms_upgrader.go:144] TidbCluster: [pingcap/basic]' pdms upgrader: check primary: http://basic-scheduling-2.basic-scheduling-peer.pingcap.svc:2379, upgradePDMSName: basic-scheduling-2, upgradePodName: basic-scheduling-2
I0813 07:57:08.289801       1 pd_ms_upgrader.go:180] Tidbcluster: [pingcap/basic]' pdms upgrader: start to choose pdms to transfer primary from members
I0813 07:57:08.289815       1 pd_ms_upgrader.go:205] Tidbcluster: [pingcap/basic]' pdms upgrader: choose pdms to transfer primary from members, targetName: basic-scheduling-0
I0813 07:57:08.289820       1 pd_ms_upgrader.go:155] TidbCluster: [pingcap/basic]' pdms upgrader: transfer pdms primary to: basic-scheduling-0
E0813 07:57:08.289834       1 pdms_api.go:67] only support TSO service, but got scheduling
I0813 07:57:08.296517       1 pd_ms_upgrader.go:161] TidbCluster: [pingcap/basic]' pdms upgrader: transfer pdms primary to: basic-scheduling-0 successfully

// `scheduling-1` will upgraded directly which the primary is `scheduling-0`
I0813 07:57:57.924827       1 pd_ms_upgrader.go:144] TidbCluster: [pingcap/basic]' pdms upgrader: check primary: http://basic-scheduling-0.basic-scheduling-peer.pingcap.svc:2379, upgradePDMSName: basic-scheduling-1, upgradePodName: basic-scheduling-1

// when upgrade `scheduling-0`, should transfer to `scheduling-2` because `scheduling-0` is primary now.
I0813 07:58:05.912621       1 statefulset.go:182] set pingcap/basic-scheduling partition to 1
I0813 07:58:05.912978       1 pd_ms_upgrader.go:144] TidbCluster: [pingcap/basic]' pdms upgrader: check primary: http://basic-scheduling-0.basic-scheduling-peer.pingcap.svc:2379, upgradePDMSName: basic-scheduling-0, upgradePodName: basic-scheduling-0
I0813 07:58:05.912998       1 pd_ms_upgrader.go:180] Tidbcluster: [pingcap/basic]' pdms upgrader: start to choose pdms to transfer primary from members
I0813 07:58:05.913011       1 pd_ms_upgrader.go:205] Tidbcluster: [pingcap/basic]' pdms upgrader: choose pdms to transfer primary from members, targetName: basic-scheduling-2
I0813 07:58:05.913022       1 pd_ms_upgrader.go:155] TidbCluster: [pingcap/basic]' pdms upgrader: transfer pdms primary to: basic-scheduling-2
E0813 07:58:05.913040       1 pdms_api.go:67] only support TSO service, but got scheduling
I0813 07:58:05.919682       1 pd_ms_upgrader.go:161] TidbCluster: [pingcap/basic]' pdms upgrader: transfer pdms primary to: basic-scheduling-2 successfully

Code changes

[x] Has Go code change

Tests

[x] Unit test
[x] Manual test
[ ] No code

Side effects

[ ] Breaking backward compatibility
[ ] Other side effects:

Related changes

[ ] Need to cherry-pick to the release branch
[ ] Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

HuSharp commented 2 months ago

@csuzhangxc PTAL, thx~

HuSharp commented 2 months ago

/run-all-tests

codecov-commenter commented 2 months ago

Codecov Report

Attention: Patch coverage is 3.50877% with 110 lines in your changes missing coverage. Please review.

Project coverage is 33.08%. Comparing base (9ef26f8) to head (d02d9b8). Report is 13 commits behind head on master.

:exclamation: There is a different number of reports uploaded between BASE (9ef26f8) and HEAD (d02d9b8). Click for more details.

HEAD has 1 upload less than BASE
| Flag | BASE (9ef26f8) | HEAD (d02d9b8) | |------|------|------| |unittest|1|0|

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #5643 +/- ## =========================================== - Coverage 61.47% 33.08% -28.39% =========================================== Files 235 219 -16 Lines 30653 30611 -42 =========================================== - Hits 18843 10127 -8716 - Misses 9920 19087 +9167 + Partials 1890 1397 -493 ``` | [Flag](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5643/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | | |---|---|---| | [e2e](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5643/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `33.08% <3.50%> (?)` | | | [unittest](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5643/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `?` | |

HuSharp commented 1 month ago

/run-all-tests

HuSharp commented 1 month ago

/run-all-tests

csuzhangxc commented 1 month ago

/run-pull-e2e-kind-across-kubernetes

csuzhangxc commented 1 month ago

/run-pull-e2e-kind-tikv-scale-simultaneously

csuzhangxc commented 1 month ago

/run-pull-e2e-kind-tngm

ti-chi-bot[bot] commented 1 month ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csuzhangxc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/pingcap/tidb-operator/blob/master/OWNERS)~~ [csuzhangxc] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

ti-chi-bot[bot] commented 1 month ago

[LGTM Timeline notifier]

Timeline:

2024-08-13 09:26:04.210377006 +0000 UTC m=+260048.913846641: :ballot_box_with_check: agreed by csuzhangxc.

HuSharp commented 1 month ago

/run-pull-e2e-kind-serial

HuSharp commented 1 month ago

/run-pull-e2e-kind

csuzhangxc commented 1 month ago

/run-pull-e2e-kind

csuzhangxc commented 1 month ago

/run-pull-e2e-kind-across-kubernetes

HuSharp commented 1 month ago

/run-pull-e2e-kind-across-kubernetes

HuSharp commented 1 month ago

/run-pull-e2e-kind

csuzhangxc commented 1 month ago

/cherry-pick release-1.6

ti-chi-bot commented 1 month ago

@csuzhangxc: new pull request created to branch release-1.6: #5709.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5643#issuecomment-2287727127): >/cherry-pick release-1.6 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.

pingcap / tidb-operator