Closed sutakori closed 2 years ago
Thank you for the questions!
run_eval_rag_re.sh
only provides the retrieval results. For text generation evaluation scores (F1, EM, BL in Table 4) , please refer to run_eval_rag_e2e.sh
run_kb_index.sh
) and the biencoder for Retriever Module in RAG (run_converter.sh
).Thanks for your reply!
So if I am not mistaken, Table 5 is from run_eval_rag_re.sh
, and Table 4&6 are from run_eval_rag_e2e.sh
, with task set as grounding&generation, is that right?
I mistakenly thought D^token-ft as DPR and D^token-rr-cls-ft as RAG, and so they are all RAG? And I am still confusing of the difference between D^token-ft and the *-rr-*.
run_eval_rag_re.sh
) and text generation evaluation (run_eval_rag_e2e.sh
) results by RAG models. Table 5 is DPR retrieval results, not RAG.rr-cls
) or pooled (rr-pl
). Please see Paper Section 3.2 and code as a reference.Ok, I've got it, thank you for your prompt reply!
I am running run_eval_rag_re.sh on BM25 baseline and seeing a much high result on retrieval results,
Settings: domain=all seg=token score=original task=grounding split=val
Additional parameters: --bm25 ../data/mdd_kb/mdd-$seg-$domain.csv
Input files are generated by predecessor scripts with same settings. Datas are generated by run_data_preprocessing.sh. Index files are generated by run_kb_index.sh. Checkpoints are generated by run_finetune_rag.sh, with DPR checkpoints generated by run_converter.sh on finetuned DPR checkpoints. (And if I am not mistaken, although required by the code, RAG checkpoints will not affect the results of run_eval_rag_re.sh with bm25 given).
So any mistake in my usage or understanding?
By the way, I am a bit confusing on the grounding span generation task (Table 4) in the paper. Does it correspond to the result of run_eval_rag_re.sh? But it dosen't contain F1, EM and BL. And does the D^token-rr-cls-ft means joint training of DPR question encoder and RAG generator, while D^token-ft use finetuned DPR directly? I would be appreciated if you could clarify my confusions.