Some questions about the checkpoint

[mingheiT]

Hello, I tried to use your checkpoint to finetune the RetroMAE_MSMARCO model, but the result is lower than the number in your paper(e.g. the MRR@10 is 0.393 in the paper, but I got 0.373). I used the hard_negs in the script, and used the same hyper-parameters. The only difference is that I use one V100 to finetune.

[staoxiao]

Thanks for your interest in our work! We use 8*GPUs to fine-tune the model, which can increase the number of in-batch negatives since we share the negatives across GPUs. Using only one GPU may impair performance. However, this result falls below my expectations. Do you use the same training and testing script in https://github.com/staoxiao/RetroMAE/tree/master/examples/retriever/msmarco?

[mingheiT]

Thank you for your explanation. I did use the same script. I got MRR@10 0.393 when I directly used RetroMAE_MSMARCO_finetune model for testing, but I can't get the same number when I use the checkpoint. This is the script I use to finetune python -m torch.distributed.launch --nproc_per_node 1 \ -m run \ --output_dir /retro-mae/pre_trained_onms/retro_mae_official_fine \ --model_name_or_path /retro-mae/pre_trained_onms/RetroMAE_MSMARCO \ --do_train \ --corpus_file /mydata/marco/data/BertTokenizer_data/corpus \ --train_query_file /mydata/marco/data/BertTokenizer_data/train_query \ --train_qrels /mydata/marco/data/BertTokenizer_data/train_qrels.txt \ --neg_file /mydata/marco/data/BertTokenizer_data/hard_negs.txt \ --query_max_len 32 \ --passage_max_len 140 \ --fp16 \ --per_device_eval_batch_size 16 \ --train_group_size 16 \ --sample_neg_from_topk 200 \ --learning_rate 2e-5 \ --num_train_epochs 4 \ --negatives_x_device \ --dataloader_num_workers 6 \ --save_steps 120000

[staoxiao]

Hi, I run the same command with one GPU (set num_train_epochs=1 for efficiency), and get MRR@10=0.383. Larger num_train_epochs may further improve the result.