Update repro log for monoBERT ranking on MS MARCO passage

[leungjch]

Reproduced on Compute Canada with 4x Tesla V100s, 4 CPUs, and 120GB RAM.

Results:

2021-09-18 16:19:18,435 - INFO - capreolus.trainer.tensorflow.train - dev metrics: MRR@10=0.345 P_1=0.226 P_10=0.063 P_20=0.036 P_5=0.103 judged_10=0.063 judged_20=0.036 judged_200=0.004 map=0.350 ndcg_cut_10=0.404 ndcg_cut_20=0.427 ndcg_cut_5=0.370 recall_100=0.810 recall_1000=0.853 recip_rank=0.354

Full training and validation logs: msmarco-psg-output.log

[crystina-z]

Hi! @leungjch thanks for running the reproduction! I noticed the MRR@10=0.345 seems to be a bit lower than expected (should be 0.35+, typically 0.354-ish). I'll check the score on my end and get back to u in a couple of days.

At the same time, could u share the version of numpy, tensorflow, and transformers? Thanks!

[leungjch]

@crystina-z Here are the versions of numpy, tensorflow, and transformers in the venv.

(capreolus-env) [j89leung@cedar1 capreolus]$ pip list | grep -E "tensorflow|numpy|transformers"
numpy                         1.21.2                           
tensorflow                    2.5.0+computecanada              
tensorflow-addons             0.13.0+computecanada             
tensorflow-datasets           4.4.0                            
tensorflow-estimator          2.5.0+computecanada              
tensorflow-hub                0.12.0+computecanada             
tensorflow-metadata           1.2.0                            
tensorflow-model-optimization 0.6.0                            
tensorflow-ranking            0.4.2                            
tensorflow-serving-api        2.6.0                            
transformers                  4.6.0                            

Here was also the sample_slurm_script.sh that I ran. I believe I only modified the file to replace the account flag #SBATCH --account=$SLURM_ACCOUNT

#!/bin/bash
#SBATCH --job-name=msmarcopsg
#SBATCH --nodes=1
#SBATCH --gres=gpu:v100l:4
#SBATCH --ntasks-per-node=1
#SBATCH --mem=120GB
#SBATCH --time=72:00:00
#SBATCH --account=$SLURM_ACCOUNT
#SBATCH --cpus-per-task=16

#SBATCH -o ./msmarco-psg-output.log

# Modify the following lines according to your setup process
module load arch/avx512 StdEnv/2018.3 java/11 python/3.7 scipy-stack
ENVDIR=$HOME/venv/capreolus-env
source $ENVDIR/bin/activate

niters=10
batch_size=16
validatefreq=$niters # to ensure the validation is run only at the end of training
decayiters=$niters   # either same with $itersize or 0
threshold=1000       # the top-k documents to rerank

python -m capreolus.run rerank.train with \
    file=docs/reproduction/config_msmarco.txt  \
    threshold=$threshold \
    reranker.trainer.niters=$niters \
    reranker.trainer.batch=$batch_size \
    reranker.trainer.decayiters=$decayiters \
    reranker.trainer.validatefreq=$validatefreq \
    fold=s1

Let me know if there's any other info you need!

[crystina-z]

moved to here