Closed thongnt99 closed 2 years ago
Hi @thongnt99 Thanks for your interest !
We get the results using a custom implementation of inverted index relying on numba for intra-query parallelization (@cadurosar ). The code is not released yet, but we plan to in the coming weeks. I am afraid this code will be indeed slow to evaluate on MS MARCO. Alternatively, you can evaluate with pyserini following this: https://github.com/castorini/anserini/blob/master/docs/experiments-msmarco-passage-splade-v2.md (you only need to create the query/document files with SPLADE weights; we also plan to release this conversion step soon, but it's not too difficult, just let us know if you need some help). Best, Thibault
Thanks. Looking forward to your future release.
Hi @thibault-formal , I wonder how the term weights were normalized? I think it was multiplied with 100 and rounded to the closet integer?
Hi @thibault-formal , I wonder how the term weights were normalized? I think it was multiplied with 100 and rounded to the closet integer?
Hi @thongnt99, you are correct, we have converted term weights to integer in that way.
Thanks. I ask because it doesn't match with the weights generated by my code.
There seems to be a small mistake in this code.
self.agg = torch.sum
should be self.agg = agg
hi @thongnt99 indeed, it was a silly mistake I made when creating the notebook ! It is fixed now, thanks for spotting it Best, Thibault
Hi, just in case it is useful, I re-trained the distilsplade_v2 and re-generated the term weights myself, it only took less than 4 minutes for searching with pyserini. (MRR@10 = 0.3744). On the same machine, it took 11 minutes with the term weights provided here.
Sorry, it took as a while, but now we have released instructions and code for evaluating on msmarco with anserini. We also released some new models :) https://github.com/naver/splade/tree/main/anserini_evaluation
Hi, thanks for your very interesting work.
Could you share how you evaluate to get the results here. Did you use inverted indexing or use this code? I am trying the later approach, but it is very slow on MSMARCO. Thank you