search

stanford-futuredata / ColBERT

ColBERT: state-of-the-art neural search (SIGIR'20, TACL'21, NeurIPS'21, NAACL'22, CIKM'22, ACL'23, EMNLP'23)
MIT License
2.68k stars 355 forks source link

Clarification on PLAID retrieval #263

Closed thibault-formal closed 5 months ago

thibault-formal commented 9 months ago

Hi @okhat and all!

In your PLAID paper (very nice btw!), I cannot understand the difference between ndocs/4 (which seems to be the number of documents after Stage 3) and k (which seems to be the number of documents to exactly re-rank in Stage 4). In the end, for how many documents true relevance scores are computed? Thx in advance! Thibault

okhat commented 9 months ago

Hey Thibault! Hope you’re well.

I can check but basically we return k, but we compute exact scores for a larger number than k

thibault-formal commented 9 months ago

Hey Omar! I hope you are well too! I see -- so basically

Just checking that I got things correctly, as I have been working on related stuff :) Thanks

okhat commented 9 months ago

This sounds right, but @santhnm2 might be able to confirm too

santhnm2 commented 9 months ago

Yes this is correct, this function is where we choose the hyperparameters according to k: https://github.com/stanford-futuredata/ColBERT/blob/main/colbert/searcher.py#L88 And here is where the number of exact scores is computed: https://github.com/stanford-futuredata/ColBERT/blob/fc3ce55d2a6c993367b176d183e0a81c6d6de8d4/colbert/search/index_storage.py#L152

thibault-formal commented 9 months ago

Perfect, thank you both for the quick answer!

thibault-formal commented 8 months ago

Hi again,

I have another (unrelated) question regarding PLAID: did you evaluate the performance on the BEIR benchmark? Could there be a performance drop (OOD) due to the approximation?

EDIT: I saw the Lotte results (apparently no drop) but I wonder if it's also true on BEIR

Thanks