search

milvus-io / milvus-tools

A data migration tool for Milvus.
Apache License 2.0
69 stars 21 forks source link

MIlvusdm issue with large scale data #13

Closed jingkl closed 3 years ago

jingkl commented 3 years ago

I tested Milvusdm with 100 million data, 10 million data, respectively, and when the amount of data is large, it simply does not run in small machines, summary: Milvusdm occupies a very high memory, please fix

shiyu22 commented 3 years ago

Thanks for your issue, I have fixed it with PR#15. And we will publish a new version later, if you want to update it immediately, you can run as follow:

$ git clone -b 1.0 https://github.com/milvus-io/milvus-tools.git
$ echo "y" | pip uninstall pymilvusdm && python setup.py install