search

OpenBMB / VisRAG

Parsing-free RAG supported by VLMs
Apache License 2.0
379 stars 29 forks source link

Reproduce retriever results #13

Closed yliuhz closed 2 weeks ago

yliuhz commented 2 weeks ago

Dear authors,

Thanks for your amazing work! I'm trying to reproduce the retriever performance in paper. I used your provided model weight of VisRAG-Ret but I think I got a quite weird reproduction result.

MRR@10 ArxivQA ChartQA DocVQA InfoVQA PlotQA SlideVQA
Reported 67.00 59.34 77.65 84.05 40.26 91.71
Reproduced 25.00 100.00 100.00 100.00 100.00 100.00

Attached is the log for command bash scripts/eval_retriever/eval.sh 512 2048 16 8 wmean causal ArxivQA,ChartQA,MP-DocVQA,InfoVQA,PlotQA,SlideVQA ./pretrained_models/VisRAG-Ret 2>&1 | tee logs/eval_r.log. Could your please help me figure out the potential reasons? Thank you very much~

eval_r.log

Best, Yue

lvshuai789 commented 2 weeks ago

I have encountered the same issue, and when I load my own trained model for testing, an error occurs, and I cannot get any results. I hope the author can address this problem and optimize the code.

tcy6 commented 2 weeks ago

I'm sorry for the bug in the code, I will fix it as soon as possible

tcy6 commented 2 weeks ago

@lvshuai789 @yliuhz We have identified the issue. Please change line 314 in src/openmatch/utils.py from dataset = datasets.load_dataset(dataset_name, 'qrels') to dataset = datasets.load_dataset(dataset_name, 'qrels')['train']. We sincerely apologize for any inconvenience our oversight may have caused you.