search

zilliztech / milvus-backup

Backup and restore tool for Milvus
Apache License 2.0
111 stars 38 forks source link

[Bug]: milvus-backup restore is failed #369

Open loxehate opened 3 days ago

loxehate commented 3 days ago

Current Behavior

./milvus-backup restore -n my_backup -s _recover

[2024/06/27 15:31:50.983 +08:00] [ERROR] [core/backup_impl_restore_backup.go:765] ["fail to bulk insert"] [db=default] [collectionName=hello_milvus1_recover] [partitionName=_default] [files="[restore-temp-restore_2024_06_27_07_31_50_795813446-default-hello_milvus1_recover/file/my_backup/binlogs/delta_log/450748688235646374/450748688235646375/450748688235646383]"] [error="unexpect file type, files=[restore-temp-restore_2024_06_27_07_31_50_795813446-default-hello_milvus1_recover/file/my_backup/binlogs/delta_log/450748688235646374/450748688235646375/450748688235646383]: importing data failed"] [errorVerbose="unexpect file type, files=[restore-temp-restore_2024_06_27_07_31_50_795813446-default-hello_milvus1_recover/file/my_backup/binlogs/delta_log/450748688235646374/450748688235646375/450748688235646383]: importing data failed\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/v2@v2.3.0-beta4.0.20240607085051-1ac5aaf24c6d/client/collection.go:39\n | github.com/milvus-io/milvus-sdk-go/v2/client.(GrpcClient).BulkInsert\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/v2@v2.3.0-beta4.0.20240607085051-1ac5aaf24c6d/client/insert.go:416\n | github.com/zilliztech/milvus-backup/core.(MilvusClient).BulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:124\n | github.com/zilliztech/milvus-backup/core.(BackupContext).executeBulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:755\n | github.com/zilliztech/milvus-backup/core.(BackupContext).executeRestoreCollectionTask.func3\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:588\n | github.com/zilliztech/milvus-backup/core.(BackupContext).executeRestoreCollectionTask.func8\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:707\n | github.com/zilliztech/milvus-backup/internal/common.(WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:1571\nWraps: (2) unexpect file type, files=[restore-temp-restore_2024_06_27_07_31_50_795813446-default-hello_milvus1_recover/file/my_backup/binlogs/delta_log/450748688235646374/450748688235646375/450748688235646383]: importing data failed\nError types: (1) withstack.withStack (2) errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(BackupContext).executeBulkInsert\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:765\ngithub.com/zilliztech/milvus-backup/core.(BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:588\ngithub.com/zilliztech/milvus-backup/core.(BackupContext).executeRestoreCollectionTask.func8\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:707\ngithub.com/zilliztech/milvus-backup/internal/common.(WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]

Expected Behavior

No response

Steps To Reproduce

No response

Environment

milvus-standalone : 2.4.1
milvus-backup: 0.4.15

Anything else?

No response

loxehate commented 3 days ago

15:56:04 开始备份，备份成功。 15:58:01 开始恢复，报错unexpect file type 日志如下： backup.log

loxehate commented 3 days ago

P7}H0Z N`KRJ_Z_LJO9%0SO 备份成功后，bucket桶中只有meta文件夹,没有binlog文件夹

wayblink commented 3 days ago

@loxehate 备份不完整？先提供下备份阶段的日志吧

loxehate commented 3 days ago

@loxehate备份不完整？ 先提供下备份阶段的日志吧

backup.log 这是k8s部署备份阶段的日志。

loxehate commented 3 days ago

我后面用docker-compose部署的milvus v2.3.15,milvus-backup: 0.4.12版本可以备份成功，也可以正常恢复

loxehate commented 3 days ago

milvus-standalone : 2.4.1 milvus-backup: 0.4.15 。不知道是不是这两个版本不兼容导致的。。。