Closed jzhoubu closed 3 years ago
Hi,
Hello, I have a query on somewhat similar lines.. In the paper, you have reported top-k accuracy, which was also defined in the paper as "the percentage of top 20/100 retrieved passages that contain the answer."
My doubt is, the top-20 accuracy should be higher than top-100 accuracy in that case, as the top documents should have higher likelihood of containing answers, and it should become lower as more distant (in the embedding space) documents are added to the list. However, in the results, the top-100 accuracy is more than top-20 accuracy, which seems counter intuitive.
Can you please help me out if with this doubt? Thank you, Krishanu.
Thanks, @vlad-karpukhin . Following your comments, I use dense_retriever
to evaluate the pre-trained model and below is the result:
Total data indexed 21015300
Data indexing completed.
Total encoded queries tensor torch.Size([3610, 768])
index search time: 3383.498359 sec.
Reading data from: /export/data/jzhoubu/workspace/DPR/data/data/wikipedia_split/psgs_w100.tsv
Matching answers in top docs...
Per question validation results len=3610
Validation results: top k documents hits [1560, 1983, 2181, 2294, 2389, 2466, 2531, 2580, 2624, 2668, 2692, 2721, 2743, 2765, 2788, 2808, 2823, 2834, 2854, 2860, 2873, 2885, 2896, 2911, 2920, 2930, 2940, 2947, 2960, 2966, 2972, 2980, 2980, 2989, 2994, 2996, 2997, 3000, 3007, 3010, 3011, 3011, 3014, 3016, 3019, 3021, 3025, 3026, 3028, 3032, 3036, 3038, 3039, 3040, 3040, 3043, 3046, 3047, 3048, 3051, 3053, 3056, 3056, 3057, 3059, 3059, 3061, 3064, 3065, 3070, 3070, 3072, 3076, 3079, 3079, 3080, 3081, 3083, 3084, 3086, 3089, 3090, 3091, 3091, 3091, 3091, 3091, 3093, 3093, 3093, 3094, 3094, 3095, 3098, 3099, 3101, 3101, 3103, 3107, 3108]
Validation results: top k documents hits accuracy [0.43213296398891965, 0.5493074792243767, 0.6041551246537397, 0.6354570637119114, 0.6617728531855955, 0.6831024930747922, 0.7011080332409972, 0.7146814404432132, 0.7268698060941828, 0.7390581717451523, 0.7457063711911357, 0.7537396121883656, 0.7598337950138504, 0.7659279778393352, 0.7722991689750692, 0.7778393351800554, 0.781994459833795, 0.7850415512465374, 0.7905817174515235, 0.7922437673130194, 0.7958448753462604, 0.7991689750692521, 0.8022160664819945, 0.8063711911357341, 0.8088642659279779, 0.8116343490304709, 0.814404432132964, 0.8163434903047091, 0.8199445983379502, 0.821606648199446, 0.8232686980609418, 0.8254847645429363, 0.8254847645429363, 0.8279778393351801, 0.8293628808864266, 0.8299168975069252, 0.8301939058171746, 0.8310249307479224, 0.8329639889196676, 0.8337950138504155, 0.8340720221606648, 0.8340720221606648, 0.8349030470914127, 0.8354570637119113, 0.8362880886426592, 0.8368421052631579, 0.8379501385041551, 0.8382271468144045, 0.838781163434903, 0.8398891966759002, 0.8409972299168975, 0.8415512465373961, 0.8418282548476455, 0.8421052631578947, 0.8421052631578947, 0.8429362880886426, 0.8437673130193906, 0.8440443213296399, 0.8443213296398892, 0.8451523545706371, 0.8457063711911358, 0.8465373961218836, 0.8465373961218836, 0.846814404432133, 0.8473684210526315, 0.8473684210526315, 0.8479224376731302, 0.8487534626038781, 0.8490304709141274, 0.850415512465374, 0.850415512465374, 0.8509695290858725, 0.8520775623268698, 0.8529085872576178, 0.8529085872576178, 0.853185595567867, 0.8534626038781163, 0.8540166204986149, 0.8542936288088643, 0.8548476454293629, 0.8556786703601108, 0.8559556786703602, 0.8562326869806094, 0.8562326869806094, 0.8562326869806094, 0.8562326869806094, 0.8562326869806094, 0.856786703601108, 0.856786703601108, 0.856786703601108, 0.8570637119113573, 0.8570637119113573, 0.8573407202216067, 0.8581717451523546, 0.8584487534626039, 0.8590027700831024, 0.8590027700831024, 0.8595567867036011, 0.8606648199445983, 0.8609418282548477]
Saved results * scores to /export/data/jzhoubu/workspace/DPR/checkpoint/retriever/multiset/bert-base-encoder.cp.test.result.top100
The top100 retrieved acc is 0.861 which is matching the reported one.
Hi @krishanudb ,
"the percentage of top 20/100 retrieved passages that contain the answer." - it doesn't mean the relative amount of passages that contains answers. Most of the questions has just a single true positive passage. Our accuracy metric actually means there is AT LEAST ONE positive (which may be a false positive) passage among 20 top results.
Hi @vlad-karpukhin Okay. Its clear now. Thanks for the explanation.
Looks like it can be closed now.
Hi, thanks for sharing the work. I tried to reproduce the retriever performance on NQ dataset (single dataset training), and below is my training command and result:
My questions are: 1) The retriever acc is surprisingly 3 points higher than that reported in the paper, is it normal? 2) I am using the default parameter on a single GPU device. If I understand it correctly, my batch size is 2 which is far smaller than yours(128) while the results are still close. Is this normal? 3) If 2) is normal, does it mean the in-batch negative samples are not that helpful? This is quite contrary to Table 3, have I missed anything?