It would be helpful to have a consistent indexing method within this package.
Currently, the benchmarking suite has a multithreaded indexer, but it's specific to LoTTE. We should abstract it and allow different files to be passed.
Currently, that code doesn't make it clear how many centroids to use. To figure that out, ColBERT recommends the square root of all embeddings that will be stored.
What I'd like to see indexing code be able to do:
Determine total number of tokens in the dataset
Calculate the number of centroids
Train k-nearest neighbors
Index the data.
We should create a library that enables this functionality instead of hiding it in the benchmark code.
It would be helpful to have a consistent indexing method within this package.
Currently, the benchmarking suite has a multithreaded indexer, but it's specific to LoTTE. We should abstract it and allow different files to be passed.
See: https://github.com/DeployQL/LintDB/blob/main/benchmarks/lotte/multiprocess_indexing.py
Currently, that code doesn't make it clear how many centroids to use. To figure that out, ColBERT recommends the square root of all embeddings that will be stored.
What I'd like to see indexing code be able to do:
We should create a library that enables this functionality instead of hiding it in the benchmark code.