search

milvus-io / milvus-tools

A data migration tool for Milvus.
Apache License 2.0
69 stars 21 forks source link

backup from milvus-cluster #34

Open parsa-ra opened 3 years ago

parsa-ra commented 3 years ago

I'm wondering how we can backup index and vector data from a Milvus cluster to an HDF5 file for HA purposes.

shiyu22 commented 3 years ago

What version of Milvus? Is there incremental data?

If you do not have incremental data, you can use the M2H function to save the original vector data, but there is no way to save the index. If you are using version 1.x, you can back up the index and original vector data by copying the milvus/db file.

parsa-ra commented 3 years ago

We are currently using version 2.0.0rc6 cluster. Is there any plan for adding this feature to future releases? indexing billion-scale datasets will take a long time and I think this is one of many required features.

shiyu22 commented 3 years ago

In Milvus 2.0 cluster, MinIO is used for data persistence of large files such as index and binary log files. And MinIO offers high-performance object storage. If you deploy Minio's distributed solution when starting the Milvus cluster, it will help you achieve HA and reliable resource storage.

parsa-ra commented 3 years ago

I wonder is it possible to clone the minio buckets(for example, using rclone) and later replace those in the newly created minio instance?

shiyu22 commented 3 years ago

Of course, you can use your own minio with the settings.