This PR introduces a new HNSW index based on the Voyager library and a sqlite based database for the dictionaries storing the document_id <-> tokens_ids associations.
This should be the default for the user as it is faster than Weaviate and does not require to use docker just a pip install.
Especially, during benches, it is not a bottleneck compared to the GPU inference anymore.
This PR introduces a new HNSW index based on the Voyager library and a sqlite based database for the dictionaries storing the document_id <-> tokens_ids associations. This should be the default for the user as it is faster than Weaviate and does not require to use docker just a pip install.
Especially, during benches, it is not a bottleneck compared to the GPU inference anymore.