Add XTR Support - Githubissues

DeployQL / LintDB

Vector Database with support for late interaction and token level embeddings.

https://www.lintdb.com/

Apache License 2.0

53 stars 2 forks source link

Add XTR Support #27

Closed mtbarta closed 5 months ago

mtbarta commented 5 months ago

This PR adds preliminary support for running XTR models in LintDB.

Tokenizer runs sentencepiece and adds an eos token for XTR.
Inverted Lists change to index codes per token for XTR. Therefore, XTR makes one less database call to score.
ProductEncoder handles quantization, with the help of new inverted list scanners and distance tables.

This touches a lot of code to help find better abstractions.

closes #11