search

pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.2k stars 490 forks source link

backup: support multiple schedules #5633

Closed BornChanger closed 1 month ago

BornChanger commented 2 months ago

What problem does this PR solve?

Close #5632

What is changed and how does it work?

In order to support multiple active schedulers, we need to co-ordinate the scheduling to make sure there is only one active backup within the same namespace. A new label is introduced in this PR, whose key is tidb.pingcap.com/backup-schedule-group. Backup schedule CRD can be assigned such a label, and the value is not empty, if it's in a multiple schedule group, otherwise, the schedule is independent. At backup schedule check time, all schedules in the same group need to be checked.

Here is a sample volume backup schedule CRD with the label specified tidb.pingcap.com/backup-schedule-group: ebs-backup-schedule-group1.

apiVersion: federation.pingcap.com/v1alpha1
kind: VolumeBackupSchedule
metadata:
  name: sche-rolling
  namespace: bc-fed-admin
  labels:
    tidb.pingcap.com/backup-schedule-group: ebs-backup-schedule-group1
spec:
  backupTemplate:
    clusters:
    - k8sClusterName: dataplane-a
      tcName: restore1-a
      tcNamespace: backup-a
    - k8sClusterName: dataplane-b
      tcName: restore1-b
      tcNamespace: backup-b
    - k8sClusterName: dataplane-c
      tcName: restore1-c
      tcNamespace: backup-c
    template:
      br:
        sendCredToTikv: false
      calcSizeLevel: disabled
      cleanPolicy: Delete
      resources: {}
      s3:
        bucket: wangle-ebs-test-us-west-2
        prefix: zm-rolling
        provider: aws
        region: us-west-2
      serviceAccount: tidb-backup-manager
      snapshotsDeleteRatio: 1
      toolImage: gcr.io/pingcap-public/zhongming/qa/br:240401173528
      volumeBackupInitJobMaxActiveSeconds: 1200
  maxReservedTime: 10m
  pause: true
  schedule: '*/10 * * * *'

This PR also provides the similar support to snapshot (non-EBS snapshot) backup schedule.

Code changes

Tests

Side effects

Related changes

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

nkg- commented 1 month ago

@BornChanger : ANy updates on this. How is the testing going. The code is ready, but I need to enhance the unit test.

ti-chi-bot[bot] commented 1 month ago

@BornChanger: Your PR was out of date, I have automatically updated it for you.

Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.
ti-chi-bot[bot] commented 1 month ago

@BornChanger: No presubmit jobs available for pingcap/tidb-operator@master

In response to [this](https://github.com/pingcap/tidb-operator/pull/5633#issuecomment-2125150348): >/test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
ti-chi-bot[bot] commented 1 month ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: WangLe1321

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/pingcap/tidb-operator/blob/master/OWNERS)~~ [WangLe1321] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
ti-chi-bot[bot] commented 1 month ago

[LGTM Timeline notifier]

Timeline:

BornChanger commented 1 month ago

/run-pull-e2e-kind-br

BornChanger commented 1 month ago

/cherry-pick release-1.5

ti-chi-bot commented 1 month ago

@BornChanger: once the present PR merges, I will cherry-pick it on top of release-1.5 in the new PR and assign it to you.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5633#issuecomment-2126811137): >/cherry-pick release-1.5 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.
BornChanger commented 1 month ago

/cherry-pick release-1.6

ti-chi-bot commented 1 month ago

@BornChanger: once the present PR merges, I will cherry-pick it on top of release-1.6 in the new PR and assign it to you.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5633#issuecomment-2126811637): >/cherry-pick release-1.6 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.
codecov-commenter commented 1 month ago

Codecov Report

Attention: Patch coverage is 0% with 29 lines in your changes are missing coverage. Please review.

Project coverage is 21.11%. Comparing base (3897095) to head (c20d828). Report is 8 commits behind head on master.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #5633 +/- ## =========================================== - Coverage 61.46% 21.11% -40.36% =========================================== Files 235 219 -16 Lines 30397 30425 +28 =========================================== - Hits 18683 6423 -12260 - Misses 9840 23092 +13252 + Partials 1874 910 -964 ``` | [Flag](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5633/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | | |---|---|---| | [e2e](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5633/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `21.11% <0.00%> (?)` | | | [unittest](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5633/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `?` | |
ti-chi-bot commented 1 month ago

@BornChanger: new pull request created to branch release-1.5: #5654.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5633#issuecomment-2126811137): >/cherry-pick release-1.5 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.
ti-chi-bot commented 1 month ago

@BornChanger: cannot checkout release-1.6: error checking out release-1.6: exit status 1. output: error: pathspec 'release-1.6' did not match any file(s) known to git

In response to [this](https://github.com/pingcap/tidb-operator/pull/5633#issuecomment-2126811637): >/cherry-pick release-1.6 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.