[Feature]: Milvus Backup from MinIO S3 bucket to External S3 (AWS/GCP)

[abijeethsaama]

Is your feature request related to a problem? Please describe.

Issue 1: Milvus does not allow to configure the object store bucket to external AWS S3 bucket. milvus.yaml config file.

  address: s3.amazonaws.com # Address of MinIO/S3
  port: 443 # Port of MinIO/S3
  accessKeyID: <access_key>
  secretAccessKey: <secret_key>
  useSSL: true # Access to MinIO/S3 with SSL
  bucketName: <bucket_name> # Bucket name in MinIO/S3
  rootPath: source # The root path where the message is stored in MinIO/S3

The above configuration still tries to connect to the minio bucket.

[2024/04/11 13:45:18.738 +00:00] [DEBUG] [config/refresher.go:71] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"]
[2024/04/11 13:45:20.069 +00:00] [ERROR] [datacoord/server.go:523] ["chunk manager init failed"] [error="Head \"https://minio:9000/<bucket_name>/\": http: server gave HTTP response to HTTPS client"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*Server).newChunkManagerFactory\n\t/root/milvus/internal/datacoord/server.go:523\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/root/milvus/internal/datacoord/server.go:335\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/root/milvus/internal/datacoord/server.go:322\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/root/milvus/internal/distributed/datacoord/service.go:129\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/root/milvus/internal/distributed/datacoord/service.go:256\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/root/milvus/cmd/components/data_coord.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/root/milvus/cmd/roles/roles.go:111"]

Issue 2: Since AWS S3 backup had issues, tried to create the backup using milvus-backup tool. This tool also creates the backup within same bucket MinIO S3 bucket.

minio:
  # cloudProvider: "minio" # deprecated use storageType instead
  storageType: "minio" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)

  address: localhost # Address of MinIO/S3
  port: 9000   # Port of MinIO/S3
  accessKeyID: minioadmin  # accessKeyID of MinIO/S3
  secretAccessKey: minioadmin # MinIO/S3 encryption string
  useSSL: false # Access to MinIO/S3 with SSL

  bucketName: "source-bucket" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
  backupBucketName: "backup_bucket" # Bucket name to store backup data. Backup data will store to 
  backupBucketName/backupRootPath

The above config copies the data within the MinIO S3 bucket. It there a way to configure to backup the milvus data from MinIO S3 to AWS S3 bucket.

Describe the solution you'd like.

@wayblink Looking for an option to backup the milvus data directly into AWS S3 or back up the data from MinIO S3 to AWS S3.

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

[wayblink]

@abijeethsaama Hi, this feature is on our long-term road map. However, won't come very soon.

[wayblink]

/assign