Hi, great work @HarshTrivedi ! I found the structure of SA(Select + Answer) model may be a little different from the paper mentioned. After reading your codes, I think the selector of SA model only selects the most K relevant passages, and these K relevant passages are not the final retrieval result. You use a classification head of your answerer to obtain the scores of K relevant passages and keep those whose scores are higher than 0.5. That means your answerer also contributes to the relevant passages retrieval task. Do I understand correctly?
Additionally, are there any SA results for full-hotpotqa and full-2wikimultihopqa datasets? And I test your SA in 2W-20k (use your serialization_dir__select_and_answer_model_selector_for_2wikimultihopqa_20k_dataset__predictions__2wikimultihopqa_dev_20k.jsonl), the Sp F1 metric is 97.35 not 99.0 in paper. Do I make some mistakes?
canghongjian
commented
11 months ago
Hi, great work @HarshTrivedi ! I found the structure of SA(Select + Answer) model may be a little different from the paper mentioned. After reading your codes, I think the selector of SA model only selects the most K relevant passages, and these K relevant passages are not the final retrieval result. You use a classification head of your answerer to obtain the scores of K relevant passages and keep those whose scores are higher than 0.5. That means your answerer also contributes to the relevant passages retrieval task. Do I understand correctly? Additionally, are there any SA results for full-hotpotqa and full-2wikimultihopqa datasets? And I test your SA in 2W-20k (use your
serialization_dir__select_and_answer_model_selector_for_2wikimultihopqa_20k_dataset__predictions__2wikimultihopqa_dev_20k.jsonl), the Sp F1 metric is97.35not99.0in paper. Do I make some mistakes?