search

zilliztech / milvus-backup

Backup and restore tool for Milvus
Apache License 2.0
135 stars 50 forks source link

[Feature]: backup object storage instance #450

Closed rere950303 closed 1 week ago

rere950303 commented 2 weeks ago

Is your feature request related to a problem? Please describe.

To prepare for DR between IDCs, I would like to use the Milvus backup feature. Therefore, I want to upload the backup files to an object storage in a different IDC from the one currently being used by Milvus.

Describe the solution you'd like.

It would be helpful if the Milvus backup configuration file allowed specifying the backup object storage address separately, so that backup files could be uploaded directly to that location. This would further clarify Milvus's philosophy of separating object storage from processing and allow for more effective DR preparedness.

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

rere950303 commented 2 weeks ago

@wayblink Dear wayblink, thank you for support!

I would like to inquire if it's possible to add a feature to set the backup object storage address in Milvus. I would appreciate it if you could check on this.

wayblink commented 2 weeks ago

@wayblink Dear wayblink, thank you for support!

I would like to inquire if it's possible to add a feature to set the backup object storage address in Milvus. I would appreciate it if you could check on this.

https://github.com/zilliztech/milvus-backup/blob/main/docs/user_guide/cross_storage.md I guess this is what you need?

rere950303 commented 2 weeks ago

@wayblink Dear wayblink, thank you for support! I would like to inquire if it's possible to add a feature to set the backup object storage address in Milvus. I would appreciate it if you could check on this.

https://github.com/zilliztech/milvus-backup/blob/main/docs/user_guide/cross_storage.md I guess this is what you need?

@wayblink Yes, I wrote an issue without knowing that the related feature was already developed. Thank you. I'm a bit confused about the cases where crossStorage needs to be set to true. Even if the storageType is the same, does it mean we need to set crossStorage to true when minio.address and backupaddress are different? Since we can have different addresses even when using the same storage service, I'd like to ask how to use crossStorage properly.

I'd like to know what the criteria are for determining if services are different.

wayblink commented 2 weeks ago

@wayblink Dear wayblink, thank you for support! I would like to inquire if it's possible to add a feature to set the backup object storage address in Milvus. I would appreciate it if you could check on this.

https://github.com/zilliztech/milvus-backup/blob/main/docs/user_guide/cross_storage.md I guess this is what you need?

@wayblink Yes, I wrote an issue without knowing that the related feature was already developed. Thank you. I'm a bit confused about the cases where crossStorage needs to be set to true. Even if the storageType is the same, does it mean we need to set crossStorage to true when minio.address and backupaddress are different? Since we can have different addresses even when using the same storage service, I'd like to ask how to use crossStorage properly.

I'd like to know what the criteria are for determining if services are different.

The crossStorage option determines how data is transferred: When set to True, data is transferred via read and write operations managed by the milvus-backup process. When set to False, data is directly copied using the storage system’s COPY API, bypassing the milvus-backup process. Use crossStorage based on whether the two addresses are accessible to each other through the COPY API.

rere950303 commented 1 week ago

I see the criterion is whether two object storages are compatible with each other's copy APIs. Thank you.