capreolus-ir / capreolus

A toolkit for end-to-end neural ad hoc retrieval
https://capreolus.ai
Apache License 2.0
95 stars 32 forks source link

Update MSMARCO reproduction log #196

Closed edanerg closed 2 years ago

edanerg commented 2 years ago

Cannot get MRR@10 = 0.35+. I have ran the experiment twice, one achieved 0.343 and another with 0.335.

I also noticed there is an issue with java version on the first run => this is probably due to compute canada changed their modules. I have modified the sample slurm script accordingly and did the second run. However the second run has a even lower MRR@10.

Reproduction logs: msmarco-psg-output.log [msmarco-psg-output-2.log](https://github.com/capreolus-ir/capreolus/files/7449055/msmarco-psg-output-2.log


Edit: Moved to this fork https://github.com/crystina-z/capreolus/blob/feature/eval%2Bptmaxp/docs/reproduction/MS_MARCO.md