Milvus Migration: migrate data to Milvus 2.x

Deprecated

This Tool is currently deprecated, please use VTS (Vector Transport Service) instead. VTS is a new tool that supports migrating data from various data sources to Milvus 2.x. It is more powerful and flexible than milvus-migration.

• Milvus
• Pinecone
• Qdrant
• Elasticsearch
• PostgreSQL
• Tencent Cloud VectorDB
• .etc

Overview

milvus-migration is a data migration tool for Milvus that supports importing Milvus 0.9.x ~ 1.x / faiss / elasticsearch7.x+ / Milvus2.3+ datas to milvus 2.x.

Architecture

Getting started

Prerequisites

• Operating System Requirements

• Software Requirements

• Data Format Support

How to use this tool?

1. Download the source code and build it:

git clone git@github.com:zilliztech/milvus-migration.git
go build

2. After building, you will obtain the milvus-migration binary file. Place this file and the migration.yaml file in the directory structure like below:

directory
---- milvus-migration
---- configs/migration.yaml

3. Milvus1.x or Faiss to Milvus2.x migration

Run the command dump to dump the source data to numpy:

./milvus-migration dump

Finally, load the numpy files to Milvus 2.x successfully by using the load command:

./milvus-migration load

4. Elasticsearch to Milvus2.x migration:

Run the command start to migration es->2.x. here start == dump && load cmd

./milvus-migration start

5. Milvus 2.3+ to Milvus2.x migration:

   ./milvus-migration start

Migration Examples (migration.yaml details)

how to learn more about using migration tool, see examples doc below:

1. es -> milvux2.x : migrate_es_doc.
2. milvus1.x -> milvux2.x : migrate_1.x_doc.
3. faiss -> milvux2.x ( Beta) : migrate_faiss_doc.
4. milvus2.x -> milvux2.x : migrate_milvus2x_doc.

How to verify migration result

When migration finished, you can use visual tool Attu or use Milvus SDK verify your new collection data rows.

• Attu
• [Milvus SDK - Get Rows](https://milvus.io/api-reference/java/v2.2.x/Collection/getCollectionStatistics().md)

Build Index and search

After the Milvus collection Data migration is completed, we can use SDK or Attu to create index and load collection for the next search operation.

• SDK - build index
• SDK - load collection
• [SDK - vector search](https://milvus.io/api-reference/java/v2.2.x/Query%20and%20Search/search().md)

Future work

• [ ] Support Redis to Milvus 2.x
• [ ] Support Mongodb to Milvus 2.x
• [ ] Support others datasource to Milvus 2.x
• [ ] Supports binary vectors

Contributing

Contributions to milvus-migration are welcome from everyone. See Contributing for details of the contribution workflow.