search

pixelogik / NearPy

Python framework for fast (approximated) nearest neighbour search in large, high-dimensional data sets using different locality-sensitive hashes.
MIT License
759 stars 152 forks source link

Multi-threading support #85

Closed Dr-Champ closed 5 years ago

Dr-Champ commented 5 years ago

Hi,

First of all, thanks for such a nice project. We've been looking for a good ANN library with persistence support for a while, so we can share the built index across multiple docker containers easily when doing horizontal scaling.

However the documentation isn't so clear on supporting multi-threading applications. How should I treat the instances of Engine class? Can they be shared across threads, or is it thread local?

Thanks

amorgun commented 5 years ago

You can't safely share an Engine instance between multiple threads without external locks.

pixelogik commented 5 years ago

Hi @Dr-Champ - when I began with NearPy I did not consider multi-threading. Shame on me. I would be more then interested in adjustments / additions which change that.

Dr-Champ commented 5 years ago

Hi @pixelogik @amorgun . Thanks for clarification. I guess it's not such a big deal. I can put sufficient locking mechanisms in place.

However the documentation could have been more clear on this point though :)