help!!Fail to fill segment backup info

jjf956123364 commented 5 months ago

[2024/01/08 14:41:38.207 +08:00] [INFO] [core/backup_context.go:100] ["{Base:0xc00033a780 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false}"] [2024/01/08 14:41:38.207 +08:00] [INFO] [core/backup_context.go:101] ["{Base:0xc00033a780 Enabled:true DebugMode:false SimpleResponse:true}"] [2024/01/08 14:41:38.214 +08:00] [INFO] [core/backup_impl_create_backup.go:541] ["collections to backup"] [collections="[col_5e9119a34ee945ad8fa472311e04f09a]"] [2024/01/08 14:41:38.214 +08:00] [INFO] [core/backup_impl_create_backup.go:225] ["start backup collection"] [db=default] [collection=col_5e9119a34ee945ad8fa472311e04f09a] [2024/01/08 14:41:38.217 +08:00] [INFO] [core/backup_impl_create_backup.go:258] ["try to get index"] [collection_name=col_5e9119a34ee945ad8fa472311e04f09a] [2024/01/08 14:41:38.223 +08:00] [INFO] [core/backup_impl_create_backup.go:278] ["field index"] [collection_name=col_5e9119a34ee945ad8fa472311e04f09a] [field_name=text] ["index info"="[{}]"] [2024/01/08 14:41:38.229 +08:00] [INFO] [core/backup_impl_create_backup.go:278] ["field index"] [collection_name=col_5e9119a34ee945ad8fa472311e04f09a] [field_name=pk] ["index info"="[{}]"] [2024/01/08 14:41:38.235 +08:00] [INFO] [core/backup_impl_create_backup.go:278] ["field index"] [collection_name=col_5e9119a34ee945ad8fa472311e04f09a] [field_name=vector] ["index info"="[{}]"] [2024/01/08 14:41:38.243 +08:00] [INFO] [core/backup_impl_create_backup.go:366] ["GetPersistentSegmentInfo before flush from milvus"] [collectionName=col_5e9119a34ee945ad8fa472311e04f09a] [segmentNumBeforeFlush=1] [2024/01/08 14:41:38.246 +08:00] [INFO] [core/backup_impl_create_backup.go:374] ["flush segments"] [collectionName=col_5e9119a34ee945ad8fa472311e04f09a] [newSealedSegmentIDs="[]"] [flushedSegmentIDs="[446758887299134173]"] [timeOfSeal=1704696098] [2024/01/08 14:41:38.248 +08:00] [INFO] [core/backup_impl_create_backup.go:387] ["GetPersistentSegmentInfo after flush from milvus"] [collectionName=col_5e9119a34ee945ad8fa472311e04f09a] [segmentNumBeforeFlush=1] [segmentNumAfterFlush=1] [2024/01/08 14:41:38.248 +08:00] [INFO] [core/backup_impl_create_backup.go:434] ["Finished fill segment"] [collectionName=col_5e9119a34ee945ad8fa472311e04f09a] [2024/01/08 14:41:38.248 +08:00] [INFO] [core/backup_impl_create_backup.go:471] ["finish build partition info"] [collectionName=col_5e9119a34ee945ad8fa472311e04f09a] [partitionNum=1] [2024/01/08 14:41:38.248 +08:00] [INFO] [core/backup_impl_create_backup.go:561] ["Finish flush all collections"] [2024/01/08 14:41:38.249 +08:00] [INFO] [core/backup_impl_create_backup.go:499] ["Begin copy data"] [collectionName=col_5e9119a34ee945ad8fa472311e04f09a] [segmentNum=1] [2024/01/08 14:41:38.249 +08:00] [ERROR] [core/backup_impl_create_backup.go:506] ["Fail to fill segment backup info"] [collection_id=446758887299134128] [partition_id=446758887299134129] [segment_id=446758887299134173] [group_id=0] [error="Get empty input path, but segment should not be empty, %sfiles/insert_log/446758887299134128/446758887299134129/446758887299134173/"] [stack="github.com/zilliztech/milvus-backup/core.(BackupContext).backupCollectionExecute\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:506\ngithub.com/zilliztech/milvus-backup/core.(BackupContext).executeCreateBackup.func2\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:567\ngithub.com/zilliztech/milvus-backup/internal/common.(WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]

备份的时候报错,能帮忙分析一下是什么原因吗？以下是我的backup.yaml

minio:
  # cloudProvider: "minio" # deprecated use storageType instead
  storageType: "minio" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure

  address: localhost # Address of MinIO/S3
  port: 9000   # Port of MinIO/S3
  accessKeyID: minioadmin  # accessKeyID of MinIO/S3
  secretAccessKey: minioadmin # MinIO/S3 encryption string
  useSSL: false # Access to MinIO/S3 with SSL
  useIAM: false
  iamEndpoint: ""

  bucketName: "a-bucket" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
  rootPath: "files" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance

  # only for azure
  backupAccessKeyID: minioadmin  # accessKeyID of MinIO/S3
  backupSecretAccessKey: minioadmin # MinIO/S3 encryption string

  backupBucketName: "a-bucket" # Bucket name to store backup data. Backup data will store to backupBucketName/backupRootPath
  backupRootPath: "backup" # Rootpath to store backup data. Backup data will store to backupBucketName/backupRootPath

wayblink commented 5 months ago

@jjf956123364 Please run ./milvus-backup check first. It is likely the backup.yaml config is not right.

jjf956123364 commented 5 months ago

(socket) [root@master milvus_backup]# ./milvus-backup check 0.4.5 (Built on 2023-12-29T02:18:57Z from Git SHA 59d9f5ac8ae99cdf6233bbf2747405054559c82c) config:backup.yaml [2024/01/09 11:07:54.401 +08:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=] [2024/01/09 11:07:54.401 +08:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log] [2024/01/09 11:07:54.408 +08:00] [INFO] [storage/minio_chunk_manager.go:131] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files] Milvus storage is empty. Please verify whether your cluster is really empty. If not, the configs(minio address, port, bucket, rootPath) may be wrong Milvus version: v2.2.11 Storage: milvus-bucket: a-bucket milvus-rootpath: files backup-bucket: a-bucket backup-rootpath: backup

jjf956123364 commented 5 months ago

@wayblink

wayblink commented 5 months ago

@jjf956123364 看下提示应该是你bucket rootpath配置的不对和milvus的配置对比一下 milvus-bucket milvus-rootpath要和milvus一样

jjf956123364 commented 5 months ago

老师，安装milvus的时候，我只配置了docker_compose.yaml文件没有配置bucket_rootpath,这个默认值是多少

jjf956123364 commented 5 months ago

@wayblink

wayblink commented 5 months ago

@jjf956123364 这样直接页面访问下localhost:9000 从minio前端页面看下就知道了估计应该是bucket：milvus-bucket rootPath：file

jjf956123364 commented 5 months ago

@wayblink 这个页面访问不了。。。我在机器上用curl也不行

wayblink commented 5 months ago

@jjf956123364 试下 localhost:9001 以及直接试下bucket：milvus-bucket rootPath：file

jjf956123364 commented 5 months ago

@wayblink 老师，我试了还是不行。

wayblink commented 5 months ago

@jjf956123364 还有个办法你进到milvus容器里那里面configs/milvus.yaml 是milvus的配置文件看下那里面的配置

jjf956123364 commented 5 months ago

@wayblink 老师，我看了bucketName是a-bucket,rootPath是files，那我之前backup.yaml配置是没问题的

wayblink commented 5 months ago

@jjf956123364 不知道哪里没对改下docker_compose.yaml 然后访问localhost:9000

SilentEchoe commented 5 months ago

你好，我也遇到了这个问题，请问是最终是怎么解决的呢？

mikeolubode commented 3 months ago

I had the same problem and it took me several days to figure it out. This was my configuration:

minio:
  bucketName: "my-s3-bucket"
  rootPath: "/my-milvus-rootfolder"

Whenever I ran ./milvus-backup check, I would get Milvus storage is empty, please verify whether your cluster is really empty And the backup would fail with fail to fill segment backup info

I figured milvus-backup was somehow not able to read my rootPath so I removed the leading / and it worked. Final config look like this

minio:
  bucketName: "my-s3-bucket"
  rootPath: "my-milvus-rootfolder"

zilliztech / milvus-backup

help!!Fail to fill segment backup info #277