search

pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.22k stars 497 forks source link

Make warmup failure/skip on corruption configurable #5622

Closed michaelmdeng closed 5 months ago

michaelmdeng commented 5 months ago

What problem does this PR solve?

Currently warmup script exits w/ error when it encounters corruption. Similar to #5621, this behavior makes sense for checking viability of backups, but conflicts w/ desire to perform a full restore and recover.

When we perform a full restore, if restore encounters corruption, warmup will fail and will cancel the actual warmup of the volumes. Desired behavior is to log the corruption but continue the warmup operation, so that, when complete, if there is only a single corrupt TiKV, we can attempt to recover using functionality added in #5585.

What is changed and how does it work?

Change updates warmup script to only exit/cancel warmup when encountering corruption when explicitly enabled. This is only enabled for check-wal-only strategy. Otherwise, we only log the corruption but continue with normal warmup.

Code changes

Tests

Side effects

Related changes

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

sre-bot commented 5 months ago

CLA assistant check
All committers have signed the CLA.

ti-chi-bot[bot] commented 5 months ago

@YuJuncen: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5622#pullrequestreview-2021168860): > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
ti-chi-bot[bot] commented 5 months ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BornChanger, YuJuncen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/pingcap/tidb-operator/blob/master/OWNERS)~~ [BornChanger] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
ti-chi-bot[bot] commented 5 months ago

[LGTM Timeline notifier]

Timeline:

csuzhangxc commented 5 months ago

/cherry-pick release-1.5

ti-chi-bot commented 5 months ago

@csuzhangxc: new pull request created to branch release-1.5: #5635.

In response to [this](https://github.com/pingcap/tidb-operator/pull/5622#issuecomment-2076874519): >/cherry-pick release-1.5 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.