search

staoxiao / RetroMAE

Codebase for RetroMAE and beyond.
Apache License 2.0
227 stars 16 forks source link

Any plan on releasing hard negative mining code? #5

Closed jordane95 closed 1 year ago

jordane95 commented 1 year ago

Hi, I noticed that you only provided your mined hard negatives of MS MARCO on google drive. But I can't find any relevant code for hard negative mining in your codebase. Would you mind release this part of code for better reproducibility? Thanks.

staoxiao commented 1 year ago

We generate the hard negatives from the ranking results of RetroMAE (fine-tuned with official bm25 negatives). You can run our inference and test commends (use train_query instead of dev_query) to generate the top-200 passages for train queries, and then delete the ground-truth passage. We will provide more codes for hard negative mining and distillation in the future.

jordane95 commented 1 year ago

Hello again, I follow your instructions and use top200 passages (gold excluded) as hard negatives to retrain the model, but only get 38.7 MRR@10, which is lower than the reported 39.3 (btw, the first stage model trained on BM25 hn gives 37.6).

All models are initialized from Shitao/RetroMAE_MARCO and trained on 4 V100 cards. Other hyper parameters are kept the same as in your README file.

I can't find out where the 0.6 gap comes from. I also can't find more training details in your paper. Could you help me replicate the results?

staoxiao commented 1 year ago

We finetune the RetroMAE-MSMARCO on 8x A100 GPUs, which maybe achieve better performance due to more negatives (enable --negatives_x_device). For convenience, we use the same negatives for our models in Tables 3 and 4 of our paper, so the first-stage model is trained based on RetroMAE instead of RetroMAE-MSMARCO. Besides, we find using the negatives retrieved by RetroMAE_MSMARCO_finetune also is good (mrr@10=0.395). Hope these information will help you.

jordane95 commented 1 year ago

Thanks! I can successfully get 39.3 with your help.