Backup txn kv data via br

3pointer commented 2 years ago

Feature Request

Is your feature request related to a problem? Please describe:

We can backup raw API data via br for tikv cluster. but for txn API data. we don't have such way to do so. and this will decrease the cluster operation and maintenance capabilities of data loss.

Describe the feature you'd like:

Backup txn API data via br.

Describe alternatives you've considered:

We can implement this feature in a simple way. we can just allow backup full range txn data first. the basic code logic is as below:

Backup

[ ] Add a new txn subcommand in backup just like raw command.
[ ] Implement the RunBackupTxn function. just like RunBackup
- [ ] Get global snapshot timestamp from pd and stop moving GC forward. just like here
- [ ] Build the backup key range. firstly we can simply transform a full backup range from utils. https://github.com/pingcap/tidb/blob/1659c3ea486d16ee161b28b7efad43e2f641f808/util/ranger/points.go#L168
- [ ] Reuse backupRanges and the default backup request can be same as tidb backup https://github.com/pingcap/tidb/blob/aa49a4c072b8cec6d3f6b60cc1453fe60ae3dd48/br/pkg/task/backup.go#L381-L391
[ ] (optional challenging work) abstract RunBackup and RunBackupTxn.

Restore

[ ] Add a new txn subcommand in restore

[ ] Distinguish the tidb backup and txn backup after read BackupMeta file by condition

func isTxnBackup(backupMeta *pb.BackupMeta) bool {
    return len(backupMeta.Schemas) == 0 && !backupMeta.IsRawKv
}

[ ] Implement the RunRestoreTxn function. just like RunRestoreRaw. because we don't need to create tables. so the most logic is as same as raw restore.
- [ ] Get backup file ranges by iterate backupMeta.Files.
- [ ] Split ranges to let region distribute evenly. https://github.com/pingcap/tidb/blob/aa49a4c072b8cec6d3f6b60cc1453fe60ae3dd48/br/pkg/task/restore_raw.go#L149
- [ ] Restore backup files but ignore start key and end key https://github.com/pingcap/tidb/blob/aa49a4c072b8cec6d3f6b60cc1453fe60ae3dd48/br/pkg/task/restore_raw.go#L160

Teachability, Documentation, Adoption, Migration Strategy:

Test

[ ] Add an integration test, just like br_rawkv.
- [ ] Add a go txn client. basically can write txn data to TiKV and check txn data is enough.
- [ ] Implement the test in run.sh and comment /run-integration-br-tests on a PR will trigger the test.

siddontang commented 2 years ago

@3pointer can you give some advices about how to add test too?

3pointer commented 2 years ago

@3pointer can you give some advices about how to add test too?

sure, I'll update it in issue description.

lidaohang commented 1 year ago

mark

fancy-lee commented 6 months ago

When executing the br backup txnor br restore txn commands, a large number of log errors are output:

[2024/05/16 08:46:31.444 +08:00] [ERROR] [client.go:226] ["[pd] request failed with a non-200 status"] [source=tikv-driver] [name=GetMinResolvedTSByStoresIDs] [url=http://idc02-jfs01-tikv-backup-003.hogpu.cc:2379/pd/api/v1/min-resolved-ts] [method=GET] [caller-id=pd-http-client] [status="404 Not Found"] [body="404 page not found\n"] [stack="github.com/tikv/pd/client/http.(clientInner).doRequest\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:226\ngithub.com/tikv/pd/client/http.(clientInner).requestWithRetry.func1\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:134\ngithub.com/tikv/pd/client/http.(clientInner).requestWithRetry\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:144\ngithub.com/tikv/pd/client/http.(client).request\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:363\ngithub.com/tikv/pd/client/http.(client).GetMinResolvedTSByStoresIDs\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/interface.go:864\ngithub.com/tikv/client-go/v2/tikv.(KVStore).getMinResolvedTSByStoresIDs\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:751\ngithub.com/tikv/client-go/v2/tikv.(KVStore).updateGlobalTxnScopeTSFromPD\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:785\ngithub.com/tikv/client-go/v2/tikv.(KVStore).updateSafeTS\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:654\ngithub.com/tikv/client-go/v2/tikv.(KVStore).safeTSUpdater\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:647"] [2024/05/16 08:46:31.444 +08:00] [ERROR] [client.go:226] ["[pd] request failed with a non-200 status"] [source=tikv-driver] [name=GetMinResolvedTSByStoresIDs] [url=http://idc02-jfs01-tikv-backup-003.hogpu.cc:2379/pd/api/v1/min-resolved-ts] [method=GET] [caller-id=pd-http-client] [status="404 Not Found"] [body="404 page not found\n"] [stack="github.com/tikv/pd/client/http.(clientInner).doRequest\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:226\ngithub.com/tikv/pd/client/http.(clientInner).requestWithRetry.func1\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:134\ngithub.com/tikv/pd/client/http.(clientInner).requestWithRetry\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:144\ngithub.com/tikv/pd/client/http.(client).request\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/client.go:363\ngithub.com/tikv/pd/client/http.(client).GetMinResolvedTSByStoresIDs\n\t/home/go21path/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240322051414-fb9e2d561b6e/http/interface.go:864\ngithub.com/tikv/client-go/v2/tikv.(KVStore).getMinResolvedTSByStoresIDs\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:751\ngithub.com/tikv/client-go/v2/tikv.(KVStore).updateSafeTS\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:673\ngithub.com/tikv/client-go/v2/tikv.(*KVStore).safeTSUpdater\n\t/home/go21path/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240322070737-05aaba6cc6f7/tikv/kv.go:647"]

What is the cause? Is there any impact on backup and restore?

3pointer commented 6 months ago

[2024/05/16 08:46:31.444 +08:00] [ERROR] [client.go:226] ["[pd] request failed with a non-200 status"] [source=tikv-driver] [name=GetMinResolvedTSByStoresIDs] [url=http://idc02-jfs01-tikv-backup-003.hogpu.cc:2379/pd/api/v1/min-resolved-ts] [method=GET] [caller-id=pd-http-client] [status="404 Not Found"] [body="404 page not found\n"]

It seems br/pd version not matched(usually means backup/restore to a old version cluster) This is a feature request issue. you can open another issue to discuss your requirement.

fancy-lee commented 6 months ago

[2024/05/16 08:46:31.444 +08:00] [ERROR] [client.go:226] ["[pd] request failed with a non-200 status"] [source=tikv-driver] [name=GetMinResolvedTSByStoresIDs] [url=http://idc02-jfs01-tikv-backup-003.hogpu.cc:2379/pd/api/v1/min-resolved-ts] [method=GET] [caller-id=pd-http-client] [status="404 Not Found"] [body="404 page not found\n"]

It seems br/pd version not matched(usually means backup/restore to a old version cluster) This is a feature request issue. you can open another issue to discuss your requirement.

https://github.com/pingcap/tidb/issues/53402

pingcap / tidb