milvus 2.3 backup files fail to restore to 2.4

OpenSourceApp-coder commented 4 weeks ago

Is there an existing issue for this?

[X] I have searched the existing issues

Environment

- Milvus version:2.3.12  升级 2.4.11
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

把2.3.12版本的的milvus备份数据还原到2.4.11的集群上失败 20240919-180914

Expected Behavior

把2.3.12版本的的milvus备份数据还原到2.4.11的集群上失败

Steps To Reproduce

把2.3.12版本的的milvus备份数据还原到2.4.11的集群上失败

Milvus Log

[2024/09/19 17:48:53.432 +08:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=] [2024/09/19 17:48:53.432 +08:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log] [2024/09/19 17:48:53.432 +08:00] [INFO] [cmd/restore.go:46] ["restore cmd input args"] [args="[]"] [2024/09/19 17:48:53.434 +08:00] [INFO] [core/backup_impl_restore_backup.go:26] ["receive RestoreBackupRequest"] [requestId=61650341-766c-11ef-9ed8-525400a6c1cb] [backupName=milvusback_10_206_150_241_20240919010101] [onlyMeta=false] [restoreIndex=true] [useAutoIndex=false] [dropExistCollection=false] [dropExistIndex=false] [skipCreateCollection=false] [collections="[]"] [CollectionSuffix=] [CollectionRenames={}] [async=false] [bucketName=] [path=] [databaseCollections=] [2024/09/19 17:48:53.434 +08:00] [INFO] [core/backup_context.go:106] ["{Base:0xc000142000 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false GcPauseEnable:true GcPauseSeconds:7200 GcPauseAddress:http://localhost:9091}"] [2024/09/19 17:48:53.434 +08:00] [INFO] [core/backup_context.go:107] ["{Base:0xc000142000 Enabled:true DebugMode:false SimpleResponse:true}"] [2024/09/19 17:48:53.434 +08:00] [INFO] [core/backup_context.go:209] ["receive GetBackupRequest"] [requestId=61653daf-766c-11ef-9ed8-525400a6c1cb] [backupName=milvusback_10_206_150_241_20240919010101] [backupId=] [bucketName=] [path=] [2024/09/19 17:48:53.439 +08:00] [INFO] [storage/minio_chunk_manager.go:151] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files] [2024/09/19 17:48:53.451 +08:00] [WARN] [storage/minio_chunk_manager.go:276] ["failed to stat object"] [path=backup/milvusback_10_206_150_241_20240919010101/meta/backup_meta.json] [error="The specified key does not exist."] [2024/09/19 17:48:53.451 +08:00] [ERROR] [core/backup_context.go:470] ["Read backup meta failed"] [path=backup/milvusback_10_206_150_241_20240919010101/meta/backup_meta.json] [error="NoSuchKey(key=backup/milvusback_10_206_150_241_20240919010101/meta/backup_meta.json)"] [stack="github.com/zilliztech/milvus-backup/core.(BackupContext).readBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_context.go:470\ngithub.com/zilliztech/milvus-backup/core.(BackupContext).GetBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_context.go:267\ngithub.com/zilliztech/milvus-backup/core.(BackupContext).RestoreBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:67\ngithub.com/zilliztech/milvus-backup/cmd.glob..func7\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/restore.go:83\ngithub.com/spf13/cobra.(Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990\ngithub.com/spf13/cobra.(Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:35\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:24\nruntime.main\n\t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/proc.go:250"] [2024/09/19 17:48:53.451 +08:00] [WARN] [core/backup_context.go:269] ["Fail to read backup"] [backupBucketName=a-bucket] [backupPath=backup/milvusback_10_206_150_241_20240919010101] [error="NoSuchKey(key=backup/milvusback_10_206_150_241_20240919010101/meta/backup_meta.json)"] [2024/09/19 17:48:53.451 +08:00] [INFO] [core/backup_context.go:301] ["finish GetBackupRequest"] [requestId=61653daf-766c-11ef-9ed8-525400a6c1cb] [backupName=milvusback_10_206_150_241_20240919010101] [backupId=] [bucketName=] [path=] [2024/09/19 17:48:53.451 +08:00] [ERROR] [core/backup_impl_restore_backup.go:84] ["fail to get backup"] [backupName=milvusback_10_206_150_241_20240919010101] [msg="not found"] [stack="github.com/zilliztech/milvus-backup/core.(BackupContext).RestoreBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:84\ngithub.com/zilliztech/milvus-backup/cmd.glob..func7\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/restore.go:83\ngithub.com/spf13/cobra.(Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990\ngithub.com/spf13/cobra.(Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:35\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:24\nruntime.main\n\t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/proc.go:250"] not found duration:0 s

Anything else?

把2.3.12版本的的milvus备份数据还原到2.4.11的集群上失败

github-actions[bot] commented 4 weeks ago

The title and description of this issue contains Chinese. Please use English to describe your issue.

OpenSourceApp-coder commented 4 weeks ago

安全扫描上面发的地址是个木马程序，大家注意

yanliang567 commented 4 weeks ago

/assign @wayblink please help to take a look /unassign

OpenSourceApp-coder commented 4 weeks ago

我不得不吐槽，这帮人多少心理上有点问题，没事改我的标题干啥。

xiaofan-luan commented 3 weeks ago

[2024/09/19 17:48:53.451 +08:00] [WARN] [core/backup_context.go:269] ["Fail to read backup"] [backupBucketName=a-bucket] [backupPath=backup/milvusback_10_206_150_241_20240919010101] [error="NoSuchKey(key=backup/milvusback_10_206_150_241_20240919010101/meta/backup_meta.json)"]

The error seems to be clear

the meta file does not exist in minio, please check where the file is located

We'd recommend to use english since there are many non chinese developer so this is the best way to make every one understand.

OpenSourceApp-coder commented 3 weeks ago

./milvus-backup restore -n milvusback --config /data/13/configs/backup.yaml --restore_index 0.4.13 (Built on 2024-05-15T07:36:05Z from Git SHA https://github.com/zilliztech/milvus-backup/commit/fbd593a1d42906626de4f9318c69b031fa12f23e) [2024/09/20 14:41:44.332 +08:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=] [2024/09/20 14:41:44.332 +08:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log] [2024/09/20 14:41:44.333 +08:00] [INFO] [cmd/restore.go:46] ["restore cmd input args"] [args="[]"] [2024/09/20 14:41:44.334 +08:00] [INFO] [core/backup_impl_restore_backup.go:26] ["receive RestoreBackupRequest"] [requestId=66be1a52-771b-11ef-8654-525400a6c1cb] [backupName=milvusback] [onlyMeta=false] [restoreIndex=true] [useAutoIndex=false] [dropExistCollection=false] [dropExistIndex=false] [skipCreateCollection=false] [collections="[]"] [CollectionSuffix=] [CollectionRenames={}] [async=false] [bucketName=] [path=] [databaseCollections=] [2024/09/20 14:41:44.334 +08:00] [INFO] [core/backup_context.go:106] ["{Base:0xc0001d5400 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false GcPauseEnable:true GcPauseSeconds:7200 GcPauseAddress:http://localhost:9091}"] [2024/09/20 14:41:44.334 +08:00] [INFO] [core/backup_context.go:107] ["{Base:0xc0001d5400 Enabled:true DebugMode:false SimpleResponse:true}"] [2024/09/20 14:41:44.335 +08:00] [INFO] [core/backup_context.go:209] ["receive GetBackupRequest"] [requestId=66be624d-771b-11ef-8654-525400a6c1cb] [backupName=milvusback] [backupId=] [bucketName=] [path=] [2024/09/20 14:41:44.340 +08:00] [INFO] [storage/minio_chunk_manager.go:151] ["minio chunk manager init success."] [bucketname=my-release] [root=files] [2024/09/20 14:41:44.350 +08:00] [WARN] [storage/minio_chunk_manager.go:276] ["failed to stat object"] [path=/backup/milvusback/meta/backup_meta.json] [error="The specified key does not exist."] [2024/09/20 14:41:44.350 +08:00] [ERROR] [core/backup_context.go:470] ["Read backup meta failed"] [path=/backup/milvusback/meta/backup_meta.json] [error="NoSuchKey(key=/backup/milvusback/meta/backup_meta.json)"] [stack="github.com/zilliztech/milvus-backup/core.(BackupContext).readBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_context.go:470\ngithub.com/zilliztech/milvus-backup/core.(BackupContext).GetBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_context.go:267\ngithub.com/zilliztech/milvus-backup/core.(BackupContext).RestoreBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:67\ngithub.com/zilliztech/milvus-backup/cmd.glob..func7\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/restore.go:83\ngithub.com/spf13/cobra.(Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990\ngithub.com/spf13/cobra.(Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:35\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:24\nruntime.main\n\t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/proc.go:250"] [2024/09/20 14:41:44.350 +08:00] [WARN] [core/backup_context.go:269] ["Fail to read backup"] [backupBucketName=my-release] [backupPath=/backup/milvusback] [error="NoSuchKey(key=/backup/milvusback/meta/backup_meta.json)"] [2024/09/20 14:41:44.350 +08:00] [INFO] [core/backup_context.go:301] ["finish GetBackupRequest"] [requestId=66be624d-771b-11ef-8654-525400a6c1cb] [backupName=milvusback] [backupId=] [bucketName=] [path=] [2024/09/20 14:41:44.350 +08:00] [ERROR] [core/backup_impl_restore_backup.go:84] ["fail to get backup"] [backupName=milvusback] [msg="not found"] [stack="github.com/zilliztech/milvus-backup/core.(BackupContext).RestoreBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:84\ngithub.com/zilliztech/milvus-backup/cmd.glob..func7\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/restore.go:83\ngithub.com/spf13/cobra.(Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990\ngithub.com/spf13/cobra.(Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:35\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:24\nruntime.main\n\t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/proc.go:250"] not found duration:0 s

在这里，我进入目录查看，文件是存在的

[root@kubespray-master01 13]# cd /opt/local-path-provisioner/pvc-4c3d17e1-c6c5-4f79-b453-c957b2bd74f7_milvus_export-my-release-minio-2/my-release/backup/milvusback/meta/backup_meta.json [root@kubespray-master01 backup_meta.json]# pwd /opt/local-path-provisioner/pvc-4c3d17e1-c6c5-4f79-b453-c957b2bd74f7_milvus_export-my-release-minio-2/my-release/backup/milvusback/meta/backup_meta.json

这个是我的操作日志

有没有微信或者QQ群之类的社区

xiaofan-luan commented 3 weeks ago

看起来这是你本地的目录，导入需要的是S3上的

OpenSourceApp-coder commented 3 weeks ago

看起来这是你本地的目录，导入需要的是S3上的

我好像发现问题了，我本地备份出来的备份文件minio会解析成json文件，但是我通过s3导入的文件，minio会识别成JSON目录，请问这个问题有什么解决方案吗？

请问怎么进行跨minio进行数据迁移？

xiaofan-luan commented 3 weeks ago

dump完成后，需要手动拷贝到新的minio上。目前暂时不支持跨minio做数据备份。

OpenSourceApp-coder commented 3 weeks ago

dump完成后，需要手动拷贝到新的minio上。目前暂时不支持跨minio做数据备份。

目前发现原因：minio部署在k8s上，启动账号权限不是root，我手动拷贝的文件无法被minio读取，授权后也不行。如何把文件移动到k8s上的权限不是root的minio集群中

milvus-io / milvus