Closed robinsongh381 closed 3 years ago
Hi @robinsongh381 ,
"as d.rank -1 on device=cuda, n_gpu=8, world size=1"
According to the training step above - you are using just DatParallel mode on 8 GPU instead of DistributedDataParallel mode on 8 gpus like:
"python -m torch.distributed.launch --nproc_per_node=8 train_dense_encoder.py ..."
Although in principle they should more or less be equivalents if using same num of gpus, we trained the biencoder using DDP mode only and unaware of specific DM mode parameters to get the on par results.
Thank you for reply I will re-train with DDP and see how it goes !
@vlad-karpukhin As you suggested, I changed training config as follows
**************** CONFIGURATION ****************
adam_betas --> (0.9, 0.999)
adam_eps --> 1e-08
batch_size --> 16
checkpoint_file_name --> dpr_biencoder
dev_batch_size --> 16
dev_file --> data/data/retriever/nq-dev.json
device --> cuda:0
distributed_world_size --> 8
do_lower_case --> True
dropout --> 0.1
encoder_model_type --> hf_bert
eval_per_epoch --> 1
fix_ctx_encoder --> False
fp16 --> False
fp16_opt_level --> O1
global_loss_buf_sz --> 150000
gradient_accumulation_steps --> 1
hard_negatives --> 1
learning_rate --> 2e-05
local_rank --> 0
log_batch_step --> 100
max_grad_norm --> 2.0
model_file --> None
n_gpu --> 1
no_cuda --> False
num_train_epochs --> 40.0
other_negatives --> 0
output_dir --> ./checkpoint/nq_best
pretrained_file --> None
pretrained_model_cfg --> bert-base-uncased
projection_dim --> 0
seed --> 12345
sequence_length --> 256
shuffle_positive_ctx --> False
train_file --> data/data/retriever/nq-train.json
train_files_upsample_rates --> None
train_rolling_loss_step --> 500
val_av_rank_bsz --> 128
val_av_rank_hard_neg --> 30
val_av_rank_max_qs --> 10000
val_av_rank_other_neg --> 30
val_av_rank_start_epoch --> 30
warmup_steps --> 1237
weight_decay --> 0.0
After training with above config, I conducted same procedures for Step 3 and Step 4 which gave out
Validation results: top k documents hits accuracy [0.4442160557268471, 0.5512161699212059, 0.6067146282973621, 0.6409729359369647, 0.666209889231472, 0.686
8790681740322, 0.7012675573826653, 0.7134863537741235, 0.7247915952951924, 0.7346123101518784, 0.7432910814205779, 0.7501427429484984, 0.7563092383236268,
0.7604202352403792, 0.7648738152335275, 0.7680712572798903, 0.7728674203494348, 0.776293251113395, 0.7799474705949526, 0.7824597464885235, 0.78451524494689
96, 0.7873701039168665, 0.7901107685280347, 0.7920520726276122, 0.7947927372387804, 0.7966198469795592, 0.7992463172319287, 0.8010734269727076, 0.803014731
072285, 0.8044992577366679, 0.8062121731186479, 0.8080392828594267, 0.8096380038826082, 0.8107799474705949, 0.8126070572113737, 0.8135206120817632, 0.81443
41669521525, 0.816032887975334, 0.8168322484869247, 0.8184309695101062, 0.8194587187392943, 0.8204864679684823, 0.8214000228388718, 0.8220851889916638, 0.8
234555212972479, 0.82414068745004, 0.8245974648852347, 0.8250542423204293, 0.8263103802672148, 0.8266529633436108, 0.8279091012903963, 0.8282516843667923,
0.8285942674431883, 0.829393627954779, 0.8301929884663698, 0.8304213771839671, 0.8308781546191618, 0.8319059038483498, 0.832591070001142, 0.833276236153934
, 0.83361881923033, 0.8340755966655248, 0.8347607628183168, 0.8353317346123101, 0.8361310951239008, 0.8364736782002969, 0.836816261276693, 0.83772981614708
23, 0.8379582048646796, 0.8386433710174718, 0.8392143428114651, 0.8400137033230558, 0.8402420920406531, 0.8403562863994518, 0.8405846751170493, 0.841041452
5522439, 0.8411556469110426, 0.84138403562864, 0.8419550074226333, 0.8425259792166268, 0.8428685622930228, 0.8437821171634121, 0.8444672833162041, 0.844695
6720338016, 0.8450382551101976, 0.8458376156217883, 0.8460660043393856, 0.8464085874157816, 0.8467511704921777, 0.8475505310037684, 0.8475505310037684, 0.8
482356971565604, 0.8491492520269499, 0.8493776407445472, 0.8499486125385406, 0.8501770012561379, 0.8504053899737353, 0.8505195843325339, 0.8507479730501313]
This seems to be much better than before ! However, I have a doubt with the result
Meanwhile I have another question, if you do not mind
I was wondering why the two figures are not the same given that it looks as if they were both trained with (presumably) batch size of 128 + 1 hard negative
If you used batch_size=16 on 8 gpus in DDP mode that actually means you had 16 8 2(+ hard neg) effective batch size which corresponds to our last line in Table 3. Also, our suggested best hyper-params settings should actually give better IR results vs those reported in the paper (we just forgot to update them in April vs original results obtained on Nov).
Hello ! First of all, thank you for sharing amazing work.
I have been trying to reproduce the Table 2 from the paper, especially for results on NQ and Trivia dataset. Below are the summary of steps I have taken and consequent result.
Step1. Dowload dataset
nq-train.json
,nq-dev.json
,trivia-train.json
andtrivia-dev.json
viadownload_data.py
Step2. Retriever training
train_dense_encoder.py
with following argumentsStep3. Retriever inference
generate_dense_embeddings.py
with following arguments./checkpoint/nq/dpr_biencoder.37.920
was the one with highest correct prediction ratio for dev setStep 4. Retriever validation against the entire set of documents
dpr_biencoder.37.920
checkpoint viadense_retriever.py
with following argumentIssue
Trivia
dataset I obtained top-100 accuracy of 64.5, again inconsistent with reported value of 85.0[Updated]
Please let me know if I did anything wrong Thank you