search

AnswerDotAI / byaldi

Use late-interaction multi-modal models such as ColPali in just a few lines of code.
Apache License 2.0
552 stars 52 forks source link

Any plans to support ColQwen2 + Similarity Maps? #47

Open asmith26 opened 1 week ago

asmith26 commented 1 week ago

Hi there,

I recently came across this tweet regarding ColQwen2 + Similarity Maps from @tonywu71https://x.com/tonywu_71/status/1846870432831402315

Just wondering if there are plans to support these?

Many thanks for any help, and this amazing work/lib!

tonywu71 commented 1 week ago

Hi @asmith26, thanks for the kind message!

Because byaldi uses colpali-engine as an internal dependency, I believe bumping the package to colpali-engine==0.3.2 should be sufficient. No need for extra code change in byaldi. You would then retrieve the embeddings for byaldi and use the new interpretability module to manually plot your own similarity maps. You can use my cookbook for an example of how I've done for ColQwen2.

But I'll let @bclavie tell you his decision on the question 😉