search

zilliztech / milvus-backup

Backup and restore tool for Milvus
Apache License 2.0
111 stars 38 forks source link

[Feature]:不同集群备份恢复报错Not found #256

Closed quanyu556 closed 5 months ago

quanyu556 commented 7 months ago

Is your feature request related to a problem? Please describe.

背景：milvus version 2.2.12，从集群A备份恢复到集群B，集群A为Milvus-standalone，集群B为k8s集群使用helm安装 问题：集群A使用./milvus_backup create -n my_backup之后，在本机minio目录下将my_backup拷贝到集群B使用的S3的backup目录下，修改使用集群B的backup.yaml之后使用命令./milvus_backup restore -n my_backup报错没有找到my_backup，使用/milvus_backup list 报错

Describe the solution you'd like.

操作步骤：：1../milvus_backup create -n my_backup，拷贝my_backup至本地 2.拷贝my_backup目录至S3的backup目录下 3.虚机执行./milvus_backup restore -n my_backup，报错faild to get backup，msg="not found" 4.虚机执行/milvus_backup list，报错fail to read backup，path=backup/my_backup，以及报错read backup meta file not exist，path=bakcup/my_backup/meta_backup_metajson

Describe an alternate solution.

集群A使用yaml如下： milvus: address: localhost port: 19530 minio: cloudProvider: "minio"

address: localhost port: 9000
accessKeyID: minioadmin
secretAccessKey: minioadmin useSSL: false useIAM: false iamEndpoint: ""

bucketName: "a-bucket" rootPath: "files"

backupAccessKeyID: minioadmin
backupSecretAccessKey: minioadmin

backupBucketName: "a-bucket" backupBucketName/backupRootPath backupRootPath: "backup"

集群B使用yaml： milvus: address: 集群B地址 port: 19530 authorizationEnabled: false

minio: cloudProvider: "aws"

address: B集群S3地址 port: 8060
accessKeyID: ak secretAccessKey: sk useSSL: false useIAM: false iamEndpoint: ""

bucketName: "bucketb" rootPath:

only for azure

backupAccessKeyID: ak
backupSecretAccessKey: sk

backupBucketName: "bucketb" backupBucketName/backupRootPath backupRootPath: "backup"

Anything else? (Additional Context)

No response

wayblink commented 6 months ago

看下你移到s3的backup 路径是完整的吗 看起来似乎是没读到meta文件

quanyu556 commented 6 months ago

看下你移到s3的backup 路径是完整的吗 看起来似乎是没读到meta文件

执行list的时候 backup_paths=[backup/,backup/my_backup]，是不是说明路径读取到了my_backup这个文件夹

wayblink commented 6 months ago

看下你移到s3的backup 路径是完整的吗 看起来似乎是没读到meta文件

执行list的时候 backup_paths=[backup/,backup/my_backup]，是不是说明路径读取到了my_backup这个文件夹

向下再查 my_backup下面应该有meta binlogs两个文件夹，两个里面都应该还有东西

quanyu556 commented 6 months ago

看下你移到s3的backup 路径是完整的吗 看起来似乎是没读到meta文件

执行list的时候 backup_paths=[backup/,backup/my_backup]，是不是说明路径读取到了my_backup这个文件夹

向下再查 my_backup下面应该有meta binlogs两个文件夹，两个里面都应该还有东西

meta下边就是四个.json的文件夹，结构和我在官网看到的一样，然后binlogs下边就是delta_log和insert_log两个文件夹

wayblink commented 6 months ago

看下你移到s3的backup 路径是完整的吗 看起来似乎是没读到meta文件

执行list的时候 backup_paths=[backup/,backup/my_backup]，是不是说明路径读取到了my_backup这个文件夹

向下再查 my_backup下面应该有meta binlogs两个文件夹，两个里面都应该还有东西

meta下边就是四个.json的文件夹，结构和我在官网看到的一样，然后binlogs下边就是delta_log和insert_log两个文件夹

backup版本用的是什么 最新的应该是5个。json文件。不过这个应该也没问题。把你报异常的日志发一下

chenjinjun commented 1 week ago

我也遇到了类似的问题， 我是把旧的单机版的备份完， 备份文件拷贝到腾讯云COS 上， 同时启动新的单机版Docker，使用同一个COS存储， 并从上传到云存储的备份还原数据， 结构和索引都恢复了， 但是数据 插入失败， 是不是 COS 和minio的存储格式有不一样

chenjinjun commented 1 week ago

以下是我的错误日志： [2024/06/25 02:49:39.326 +00:00] [INFO] [core/backup_impl_restore_backup.go:743] ["execute bulk insert"] [db=default] [collection=mt_test] [partition=_default] [files="[milvusback/mbackup/binlogs/insert_log/450688120260217718/450688120260217719/450688120260217949/,]"] [endTime=0] [2024/06/25 02:49:40.664 +00:00] [INFO] [core/backup_impl_restore_backup.go:794] ["bulkinsert task state"] [id=450699564969195873] [state=0] [state="{\"ID\":0,\"State\":0,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"\",\"progress_percent\":\"0\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":1719283780}"] [progress=0] [currentTimestamp=1719283780] [lastUpdateTime=1719283780] [2024/06/25 02:49:45.671 +00:00] [INFO] [core/backup_impl_restore_backup.go:794] ["bulkinsert task state"] [id=450699564969195873] [state=1] [state="{\"ID\":0,\"State\":1,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"failed_reason\":\"misaligned binlog count, field450688120260217949:1, field0:2: importing data failed\",\"progress_percent\":\"0\"},\"CollectionID\":0,\"SegmentIDs\":null,\"CreateTs\":1719283780}"] [progress=0] [currentTimestamp=1719283785] [lastUpdateTime=1719283780] [2024/06/25 02:49:45.671 +00:00] [ERROR] [core/backup_impl_restore_backup.go:775] ["fail or timeout to bulk insert"] [error="bulk insert fail, info: misaligned binlog count, field450688120260217949:1, field0:2: importing data failed"] [errorVerbose="bulk insert fail, info: misaligned binlog count, field450688120260217949:1, field0:2: importing data failed\n(1) attached stack trace\n -- stack trace:\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).watchBulkInsertState\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:804\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:773\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:588\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:668\n | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:1571\nWraps: (2) bulk insert fail, info: misaligned binlog count, field450688120260217949:1, field0:2: importing data failed\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [taskId=450699564969195873] [targetCollectionName=mt_test] [partitionName=_default] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:775\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:588\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:668\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"] [2024/06/25 02:49:45.671 +00:00] [ERROR] [core/backup_impl_restore_backup.go:668] ["fail to bulk insert to partition"] [backup_db_name=default] [backup_collection_name=default] [target_db_name=default] [target_collection_name=mt_test] [partition=_default] [error="bulk insert fail, info: misaligned binlog count, field450688120260217949:1, field0:2: importing data failed"] [errorVerbose="bulk insert fail, info: misaligned binlog count, field450688120260217949:1, field0:2: importing data failed\n(1) attached stack trace\n -- stack trace:\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).watchBulkInsertState\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:804\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:773\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:588\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:668\n | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:1571\nWraps: (2) bulk insert fail, info: misaligned binlog count, field450688120260217949:1, field0:2: importing data failed\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:668\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"] [2024/06/25 02:49:45.671 +00:00] [ERROR] [core/backup_impl_restore_backup.go:357] ["executeRestoreCollectionTask failed"] [TargetDBName=default] [TargetCollectionName=mt_test] [error="bulk insert fail, info: misaligned binlog count, field450688120260217949:1, field0:2: importing data failed"] [errorVerbose="bulk insert fail, info: misaligned binlog count, field450688120260217949:1, field0:2: importing data failed\n(1) attached stack trace\n -- stack trace:\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).watchBulkInsertState\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:804\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:773\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:588\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:668\n | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:1571\nWraps: (2) bulk insert fail, info: misaligned binlog count, field450688120260217949:1, field0:2: importing data failed\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:357\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"] [2024/06/25 02:49:45.671 +00:00] [ERROR] [core/backup_impl_restore_backup.go:321] ["execute restore collection fail"] [backupId=140eb65f-3297-11ef-b0d0-0242ac140002] [error="workerpool: execute job bulk insert fail, info: misaligned binlog count, field450688120260217949:1, field0:2: importing data failed"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).RestoreBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:321\ngithub.com/zilliztech/milvus-backup/cmd.glob..func7\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/restore.go:83\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:35\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:24\nruntime.main\n\t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/proc.go:250"]