search

microsoft / SPTAG

A distributed approximate nearest neighborhood search (ANN) library which provides a high quality vector index build, search and distributed online serving toolkits for large scale vector search scenario.
MIT License
4.83k stars 583 forks source link

Pip package #219

Open mastertilla opened 3 years ago

mastertilla commented 3 years ago

As part of my work with the Microsoft Research team, we developed a project that make use of SPTAG to index 30000 datapoints for better search results. We found working with SPTAG challenging for multiple reasons, including the long build time as well as issues with boost versioning in the Dockerfile. To solve this, we decided to create a SPTAG pip installable package that we would like to contribute to the repo. The package is currently stored in a private repository, but we would like to share it with the community in order to reduce the entry barrier when implementing SPTAG in production.

The solution I propose is to provide the pip installable package for indexing in Pypi. This solution also includes a Dockerfile that takes care of all associated UNIX dependencies. However, all cmake and boost dependencies are not required anymore, as they are comprised within the package.

Before launching a PR, I wanted to get see how the main contributors feel about this (since no similar solution has been provided even though this issue was already raised in 2019). Happy to discuss any technical questions about the package.

mastertilla commented 3 years ago

@bkarsin any thoughts on this?

Thanks!

bkarsin commented 3 years ago

This sounds like a good way to make SPTAG more accessible and lower the barrier to entry for new users. I'm afraid I have only worked on the GPU acceleration aspects of SPTAG, though. Maybe someone more involved in full project can comment on this, @MaggieQi any thoughts?

mastertilla commented 3 years ago

Any news on this? @MaggieQi

masterkunm commented 3 years ago

Hi, I am having problems with downloading the docker for macOS, and just see this post. Is SPTAG a pip installing package yet?

EmElleE commented 2 years ago

The SPTAG pip package fails to import sptag.SPTAG I believe this has to do with python 3.8 and swig.