search

lancedb / lance

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..
https://lancedb.github.io/lance/
Apache License 2.0
3.79k stars 207 forks source link

IVF_SQ index #909

Open fferroni opened 1 year ago

fferroni commented 1 year ago

Hello, Do you support or plan on supporting SQ compression? I see you support PQ compression. FAISS also supports milder SQ compression which for my use-cases works better. Wondering if this is on the road map. If not, where could I look to adding it?

eddyxu commented 1 year ago

Hey, @fferroni

Thanks for looking into lance. We would love to and open to supporting more Faiss index types in Lance. Our current rodmap is improving IVF_PQ indexing perf and caching behavior, as well as implementing a performant DiskANN, as near term roadmap. IVF_SQ or IVF_PQfs would be nice to have, we just don't have a good estimation when we can get on to it yet.

If you are interested, you can take a look at https://github.com/lancedb/lance/tree/main/rust/src/index/vector directory, with our existing index implementations. Please take a look, we will be super happy to accept contributions from the community. Even giving us some rough guidance and estimations would help us to plan on the feature too.

Feel free to leave feedback here or shot us an email at contact@lancedb.com.