add Engine.store_many_vectors() for uploading to Redis using pipeline

pixelogik / NearPy

Python framework for fast (approximated) nearest neighbour search in large, high-dimensional data sets using different locality-sensitive hashes.

MIT License

759 stars 152 forks source link

add Engine.store_many_vectors() for uploading to Redis using pipeline #71

Closed xieqihui closed 6 years ago

xieqihui commented 6 years ago

Add store_many_vectors() method for RedisStorage(). This method uses redis.pipeline() to store key-values in a batch. With this method, the speed of storing hashes is significantly increased for problems with a large amount of hashes.

This method has been applied well in my project where the number of hashes is around 3000.

pixelogik commented 6 years ago

@xieqihui Nice, thanks!

amorgun commented 6 years ago

I recommend adding some tests.

pixelogik commented 6 years ago

@amorgun Yeah, on it.