search

dvlab-research / MGM

Official repo for "Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models"
Apache License 2.0
3.11k stars 277 forks source link

Use of ocr in Evaluation #95

Open bruceisme opened 2 months ago

bruceisme commented 2 months ago

In Appendix A's Image-text Data Collection, mention "It is important to note that the OCR detector is utilized solely for generating enriched data and is not employed during testing ". But the textvqa scripts is using llava_textvqa_val_v051_ocr.jsonl which has ocr. So have you ever test a version without ocr in textvqa, was it worse than llava_textvqa_val_v051_ocr.jsonl ? can we understand that model could get better result with ocr input? 5a6ce66bec9d6006880fe0724c32204

yanwei-li commented 2 months ago

Hi, the word in Appendix A means that we do not perform an extra PaddleOCR detector for evaluation. For the TextVQA, we keep the OCR Token with that in LLaVA. It should have a worse result without the original OCR tokens.