search

open-compass / VLMEvalKit

Open-source evaluation toolkit of large vision-language models (LVLMs), support ~100 VLMs, 40+ benchmarks
https://huggingface.co/spaces/opencompass/open_vlm_leaderboard
Apache License 2.0
1.27k stars 182 forks source link

Custom dataset is not considered MCQ by models #336

Closed zodiacg closed 1 month ago

zodiacg commented 3 months ago

VLMEvalKit version: commit 8e0aace0504d952a25e310a1de66a32c2c1476f1

I added a custom MCQ format dataset to LMUData directory. It is successfully loaded and shows "UserWarning: Will assume unsupported dataset **** as a Custom MCQ dataset". However the model is not correctly called to build a MCQ prompt. After checking it seems DATASET_TYPE function only returns correct evaluation types for supported datasets (i.e. listed in its DATASET_URL).

Since there IS CustomMCQDataset and CustomVQADataset type, is it possible to return the correct dataset type for prompt building?

Building a tsv file is very straight forward without the need of modifications to VLMEvalKit codes, so I hope it can be better supported.

kennymckormick commented 2 months ago

Hi, @zodiacg , That makes sense, we plan to treat all custom benchmarks as MCQ benchmarks and will modify the code accordingly.

kennymckormick commented 1 month ago

Feature Supported.