stanford-futuredata / ColBERT

ColBERT: state-of-the-art neural search (SIGIR'20, TACL'21, NeurIPS'21, NAACL'22, CIKM'22, ACL'23, EMNLP'23)
MIT License
2.67k stars 355 forks source link

Results on BEIR #341

Open hfawaz opened 1 month ago

hfawaz commented 1 month ago

Hello thanks for this!

Based on the ColbertV2 paper: https://arxiv.org/pdf/2112.01488

In table 5, you have very low results on NQ for example compared to some models in the leaderboard for retrieval: https://huggingface.co/spaces/mteb/leaderboard

Is there a reason behind this?

If single-vector retrievers from mteb leaderboard are better at most datasets, why use the multi-vector ?

Thanks