zilliztech / milvus-backup

Backup and restore tool for Milvus
Apache License 2.0
111 stars 38 forks source link

[DOCS]: milvus-backup minio 参数详解 #244

Closed gaoyuan5251 closed 5 months ago

gaoyuan5251 commented 7 months ago

Documentation Link

https://milvus.io/docs/milvus_backup_cli.md https://raw.githubusercontent.com/zilliztech/milvus-backup/master/configs/backup.yaml

Describe the problem

背景:从milvus 2.3.1-standalone迁移数据到,milvus2.3.3-standalone,用到milvus-backup来进行迁移。

minio 相关的参数都怎么填呢? minio: cloudProvider: "minio" # remote cloud storage provider: s3, gcp, aliyun, azure

address: localhost # Address of MinIO/S3 port: 9000 # Port of MinIO/S3 --这个必须要9000端口么? accessKeyID: minioadmin # accessKeyID of MinIO/S3 --这个账号密码是minio cluster的登陆密码? secretAccessKey: minioadmin # MinIO/S3 encryption string useSSL: false # Access to MinIO/S3 with SSL useIAM: false iamEndpoint: ""

bucketName: "a-bucket" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance rootPath: "files" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance

only for azure backupAccessKeyID: minioadmin # accessKeyID of MinIO/S3 backupSecretAccessKey: minioadmin # MinIO/S3 encryption string

backupBucketName: "a-bucket" # Bucket name to store backup data. Backup data will store to backupBucketName/backupRootPath backupRootPath: "backup" # Rootpath to store backup data. Backup data will store to backupBucketName/backupRootPath

Describe the improvement

No response

Anything else?

我确定这个文件是存在的。 、 [root@w milvus_backup]# ./milvus-backup restore -n my_backup 0.4.2 (Built on 2023-11-08T03:28:20Z from Git SHA 84e72fc70b0492591bf7e6081c10bd89887f0bae) [2023/11/28 15:30:53.971 +08:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=] [2023/11/28 15:30:53.971 +08:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log] [2023/11/28 15:30:53.971 +08:00] [INFO] [cmd/restore.go:42] ["restore cmd input args"] [args="[]"] [2023/11/28 15:30:53.972 +08:00] [INFO] [core/backup_impl_restore_backup.go:26] ["receive RestoreBackupRequest"] [requestId=102d7c9a-8dc0-11ee-8885-fa163e56ae07] [backupName=my_backup] [onlyMeta=false] [restoreIndex=false] [collections="[]"] [CollectionSuffix=] [CollectionRenames={}] [async=false] [bucketName=] [path=] [databaseCollections=] [2023/11/28 15:30:53.972 +08:00] [INFO] [core/backup_context.go:97] ["{Base:0xc0004bf680 MaxSegmentGroupSize:2147483648 BackupParallelism:2 RestoreParallelism:2 BackupCopyDataParallelism:128 KeepTempFiles:false}"] [2023/11/28 15:30:53.972 +08:00] [INFO] [core/backup_context.go:98] ["{Base:0xc0004bf680 Enabled:true DebugMode:false SimpleResponse:true}"] [2023/11/28 15:30:53.972 +08:00] [INFO] [core/backup_context.go:165] ["receive GetBackupRequest"] [requestId=102d94a1-8dc0-11ee-8885-fa163e56ae07] [backupName=my_backup] [backupId=] [bucketName=] [path=] [2023/11/28 15:30:53.982 +08:00] [INFO] [storage/minio_chunk_manager.go:130] ["minio chunk manager init success."] [bucketname=hll-milvus] [root=files] [2023/11/28 15:30:53.997 +08:00] [WARN] [core/backup_context.go:408] ["read backup meta file not exist"] [path=backup/my_backup/meta/backup_meta.json] [] [2023/11/28 15:30:53.997 +08:00] [INFO] [core/backup_context.go:245] ["finish GetBackupRequest"] [requestId=102d94a1-8dc0-11ee-8885-fa163e56ae07] [backupName=my_backup] [backupId=] [bucketName=] [path=] [2023/11/28 15:30:53.997 +08:00] [ERROR] [core/backup_impl_restore_backup.go:80] ["fail to get backup"] [backupName=my_backup] [msg="not found"] [stack="github.com/zilliztech/milvus-backup/core.(BackupContext).RestoreBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:80\ngithub.com/zilliztech/milvus-backup/cmd.glob..func6\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/restore.go:79\ngithub.com/spf13/cobra.(Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990\ngithub.com/spf13/cobra.(Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:28\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:24\nruntime.main\n\t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/proc.go:250"] not found duration:0 s [root@w milvus_backup]# cat backup/my_backup/meta/backup_meta.json {"id":"864a459f-8dbf-11ee-ab42-00163e1619a7","state_code":2,"start_time":1701156422785,"end_time":1701156431103,"name":"my_backup","backup_timestamp":1701156422785,"size":483534702,"milvus_version":"v2.3.1"}[root@w***** milvus_backup]# ls backup/my_backup/ binlogs/ meta/

wayblink commented 7 months ago

@gaoyuan5251 端口是你当前操作milvus所依赖的minio的端口,minioadmin是默认的minio登陆账号密码,这俩都是依据你的minio部署来的。

新旧集群是否用的是同一个minio,backup是把数据备份到旧milvus依赖的minio,restore是从新milvus依赖的minio里读取备份文件,如果不是同一个minio,需要在两者之间手动拷贝文件