zilliztech / milvus-backup

Backup and restore tool for Milvus
Apache License 2.0
111 stars 38 forks source link

[Bug]: milvus-backup is always in a task state #363

Open ZYWNB666 opened 1 week ago

ZYWNB666 commented 1 week ago

image

Current Behavior

./milvus-backup create -n backup_test The ip addresses entered in the backup configuration file are the milvus and minio addresses of the official environment I then copied the backup files to the test environment

Finally, change all the connection addresses in the configuration file to the ip address of the test environment ./milvus-backup restore --restore_index -n backup_test -s _recover

Perform the restore operation, and the data seems to be restored, but the tool remains in this state Although the data has been restored, I am not sure whether I did the right thing because of this state My configuration file is as follows

# Configures the system log output.
log:
  level: info # Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
  console: true # whether print log to console
  file:
    rootPath: "backup.log"

http:
  simpleResponse: true

# milvus proxy address, compatible to milvus.yaml
milvus:
  #address: 10.100.24.89
  address: 10.99.122.187 #staging
  port: 19530
  authorizationEnabled: false
  # tls mode values [0, 1, 2]
  # 0 is close, 1 is one-way authentication, 2 is two-way authentication.
  tlsMode: 0
  user: "root"
  password: "Milvus"

# Related configuration of minio, which is responsible for data persistence for Milvus.
minio:
  # cloudProvider: "minio" # deprecated use storageType instead
  storageType: "minio" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)

  #address: 10.98.33.114 # Address of MinIO/S3
  address: 10.109.241.231 # Address of MinIO/S3 staging
  port: 9000   # Port of MinIO/S3
  accessKeyID: minioadmin  # accessKeyID of MinIO/S3
  secretAccessKey: minioadmin # MinIO/S3 encryption string
  useSSL: false # Access to MinIO/S3 with SSL
  useIAM: false
  iamEndpoint: ""

  #bucketName: "milvus-bucket" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
  bucketName: "milvus-bucket" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
  rootPath: "file" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance

  # only for azure
  backupAccessKeyID: minioadmin  # accessKeyID of MinIO/S3
  backupSecretAccessKey: minioadmin # MinIO/S3 encryption string

  backupBucketName: "milvus-bucket-backup" # Bucket name to store backup data. Backup data will store to backupBucketName/backupRootPath
  backupRootPath: "backup" # Rootpath to store backup data. Backup data will store to backupBucketName/backupRootPath

backup:
  maxSegmentGroupSize: 20G

  parallelism: 
    # collection level parallelism to backup
    backupCollection: 4
    # thread pool to copy data. reduce it if blocks your storage's network bandwidth
    copydata: 128
    # Collection level parallelism to restore
    restoreCollection: 2

  # keep temporary files during restore, only use to debug 
  keepTempFiles: false

  # Pause GC during backup through Milvus Http API. 
  gcPause:
    enable: false
    seconds: 7200
    address: http://localhost:9091

Expected Behavior

It should have been successful, but it didn't return any value

Steps To Reproduce

./milvus-backup create -n backup_test
./milvus-backup restore --restore_index -n backup_test -s _recover

Environment

ubuntu 20.4

Anything else?

No response

zhuwenxing commented 1 week ago

when progress is 70%, it is in the stage of building index.

ZYWNB666 commented 1 week ago

当进度达到70%时,进入建设指标阶段。

What will be done at this stage? I don't understand, because my data has already been transferred, what else will be done.

This stage has been going on for five hours, and I have to suspect a bug

zhuwenxing commented 1 week ago

because your command is ./milvus-backup restore --restore_index -n backup_test -s _recover so only when index building is finished , this task will be considered as success.

Can you provide some logs of Milvus? We need to check why the building index costs so much time.

ZYWNB666 commented 1 week ago

This is backup's log backup (2).log

zhuwenxing commented 1 week ago

This is backup's log backup (2).log

Milvus log is also needed

ZYWNB666 commented 1 week ago

This is backup's log backup (2).log

Milvus log is also needed

Probably because there is too much data and the index is too complicated

However, I ran into another problem, the recovered data was twice the data before the backup

image

ZYWNB666 commented 1 week ago

This is backup's log backup (2).log

Milvus log is also needed

Probably because there is too much data and the index is too complicated

However, I ran into another problem, the recovered data was twice the data before the backup

image

Is it because the Celebrity was already loaded into memory when I backed it up that there were two copies?