sachin-japate commented 1 week ago

What problem does this PR solve?

This PR is addressing the issue issue-5657

Adding support for the logRestoreStartTs option in the PITR restore spec.

What is changed and how does it work?

With this update, the Point-in-Time Recovery (PITR) restore mode allows users to apply PITR logs within a specified time range. Specifically, users can define the start and end timestamps using the logRestoreStartTs(start timestamp) and pitrRestoredTs(end timestamp) options in the restore spec, respectively. This enhancement enables more precise control over the restoration process, allowing users to restore data to an exact point in time by specifying the interval of PITR logs to be applied.

After this change the user can use a combination of pitrFullBackupStorageProvider and pitrRestoredTs, to restore the full backup snapshot and apply PITR logs. or else a combination of the logRestoreStartTs and pitrRestoredTs options in the restore spec to apply PITR logs within a specified time.

If the user fails to specify either the pitrFullBackupStorageProvider or logRestoreStartTs option in the restore specification, the restore job will result in an error.

Code changes

[x] Has Go code change
[ ] Has CI related scripts change

Tests

[ ] Unit test
[ ] E2E test
[x] Manual test
PITR up to mentioned restoreTs along with full backup restore.
1. Create a TiDBCluster
2. Generate some data in the database (I used Sysbench prepare subcommand)
3. Start the PITR log Backup, refer Log Backup
4. Create a full backup, Full backup
5. Now create another TiDBCluster to test the backup restoration.
6. Follow the steps to restore full backup and apply PITR logs to the specified point(timestamp). refer Restore full backup + apply logs

Apply PITR logs within a specified time range.

Create a restore-pitr.yaml file as follows, then use kubectl apply to apply the yaml.

`

restore-pitr.yaml

 ---
 apiVersion: pingcap.com/v1alpha1
 kind: Restore
 metadata:
    name: demo3-restore-gcs
    namespace: restore-test
spec:
  restoreMode: pitr
 br:
     cluster: demo3
     clusterNamespace: test3
 gcs:
     projectId: ${project_id}
     secretName: gcs-secret
     bucket: my-bucket
     prefix: my-log-backup-folder-pitr
  logRestoreStartTs: "2022-10-10T16:21:00+08:00"
  pitrRestoredTs: "2022-10-10T17:21:00+08:00"

`

Side effects

[ ] Breaking backward compatibility: NONE
[ ] Other side effects: NONE

Related changes

[ ] Need to cherry-pick to the release branch
[x] Need to update the documentation

Release Notes

NONE

sre-bot commented 1 week ago

All committers have signed the CLA.

ti-chi-bot[bot] commented 1 week ago

Welcome @sachin-japate! It looks like this is your first PR to pingcap/tidb-operator 🎉

BornChanger commented 1 week ago

/cc @WangLe1321 ptal

ti-chi-bot[bot] commented 1 week ago

@BornChanger: GitHub didn't allow me to request PR reviews from the following users: ptal.

Note that only pingcap members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5667#issuecomment-2185985806): >/cc @WangLe1321 ptal Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

BornChanger commented 1 week ago

/cc @kennytm

kennytm commented 1 week ago

LGTM in principle, but we could do an enhancement. In BR On-Prem we additionally checked that --start-ts and --full-backup-storage cannot both exist. I think here we should also ensure exactly one of pitrFullBackupStorageProvider and logRestoreStartTs is provided, not both or zero.

sachin-japate commented 1 week ago

LGTM in principle, but we could do an enhancement. In BR On-Prem we additionally checked that --start-ts and --full-backup-storage cannot both exist. I think here we should also ensure exactly one of pitrFullBackupStorageProvider and logRestoreStartTs is provided, not both or zero.

@kennytm Added validations in the restore spec as you requested.

ti-chi-bot[bot] commented 1 week ago

@kennytm: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5667#pullrequestreview-2138250717): > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

ti-chi-bot[bot] commented 1 week ago

@RidRisR: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5667#pullrequestreview-2140333659): > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

ti-chi-bot[bot] commented 1 week ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kennytm, RidRisR, WangLe1321

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/pingcap/tidb-operator/blob/master/OWNERS)~~ [WangLe1321] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

ti-chi-bot[bot] commented 1 week ago

[LGTM Timeline notifier]

Timeline:

2024-06-26 03:58:27.992275976 +0000 UTC m=+778434.477764809: :ballot_box_with_check: agreed by WangLe1321.

WangLe1321 commented 1 week ago

/run-pull-e2e-kind-br

codecov-commenter commented 1 week ago

Codecov Report

Attention: Patch coverage is 33.33333% with 4 lines in your changes missing coverage. Please review.

Project coverage is 33.30%. Comparing base (9ef26f8) to head (5335b3c).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #5667 +/- ## =========================================== - Coverage 61.47% 33.30% -28.17% =========================================== Files 235 219 -16 Lines 30653 30433 -220 =========================================== - Hits 18843 10136 -8707 - Misses 9920 18941 +9021 + Partials 1890 1356 -534 ``` | [Flag](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5667/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | | |---|---|---| | [e2e](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5667/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `33.30% <33.33%> (?)` | | | [unittest](https://app.codecov.io/gh/pingcap/tidb-operator/pull/5667/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `?` | |

WangLe1321 commented 1 week ago

/run-pull-e2e-kind /run-pull-e2e-kind-across-kubernetes /run-pull-e2e-kind-basic /run-pull-e2e-kind-serial

WangLe1321 commented 1 week ago

/run-pull-e2e-kind /run-pull-e2e-kind-across-kubernetes

WangLe1321 commented 1 week ago

/run-pull-e2e-kind

WangLe1321 commented 1 week ago

/run-pull-e2e-kind-across-kubernetes

WangLe1321 commented 1 week ago

/run-pull-e2e-kind-basic

WangLe1321 commented 1 week ago

/run-pull-e2e-kind-serial

csuzhangxc commented 1 week ago

/cherry-pick release-1.5

csuzhangxc commented 1 week ago

/cherry-pick release-1.6

ti-chi-bot commented 1 week ago

@csuzhangxc: new pull request created to branch release-1.5: #5668.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5667#issuecomment-2191115385): >/cherry-pick release-1.5 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.

ti-chi-bot commented 1 week ago

@csuzhangxc: new pull request created to branch release-1.6: #5669.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5667#issuecomment-2191115654): >/cherry-pick release-1.6 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.

pingcap / tidb-operator

Adding support for logRestoreStartTs option in PITR restore spec #5667

What problem does this PR solve?

What is changed and how does it work?

Code changes

Tests

PITR up to mentioned restoreTs along with full backup restore.

Apply PITR logs within a specified time range.

restore-pitr.yaml

Side effects

Related changes

Release Notes

[LGTM Timeline notifier]

Codecov Report