DocVQA_TEST和InfoVQA_TEST无法评测

open-compass / VLMEvalKit

Open-source evaluation toolkit of large vision-language models (LVLMs), support 160+ VLMs, 50+ benchmarks

https://huggingface.co/spaces/opencompass/open_vlm_leaderboard

Apache License 2.0

1.34k stars 188 forks source link

DocVQA_TEST和InfoVQA_TEST无法评测 #504

Closed helloworld01001 closed 3 weeks ago

helloworld01001 commented 1 month ago

你好！我在评测DocVQA_TEST和InfoVQA_TEST时模型生成了prediction，但是会报这个错误，我打开tsv文件也没有answer，请问你是怎么进行评测的？ File "/share/project/open-compass/VLMEvalKit/vlmeval/dataset/image_vqa.py", line 46, in evaluate assert 'answer' in data and 'prediction' in data AssertionError

kennymckormick commented 1 month ago

Hi, @helloworld01001 ,

对于你提到的这两个数据集，由于数据文件中不包含 GT answer，因此代码库目前仅支持推理，不支持输出评测精度。推理文件应输出在 {modelname}{dataset_name}.xlsx 中