search

open-compass / VLMEvalKit

Open-source evaluation toolkit of large vision-language models (LVLMs), support ~100 VLMs, 40+ benchmarks
https://huggingface.co/spaces/opencompass/open_vlm_leaderboard
Apache License 2.0
1.27k stars 183 forks source link

Adding new dataset: worldmedQA-V to VLMEvalKit #525

Closed shan23chen closed 19 hours ago

shan23chen commented 2 weeks ago

Hello everyone,

Thank you for open-sourcing such a great evaluation repo!

I’m Shan Chen, primarily working in the field of health AI at Harvard.

We’ve recently developed a multimodal evaluation dataset, cleaning up medical exam data from four countries (Japan, Israel, Spain, and Brazil) over several years.

This work is being submitted to NAACL 2025, and the format of our data is the VLMEvalKit MCQ format (identical to MMbench, etc.). We hope it can be integrated into the official repo. We can provide you guys the huggingface dataset link for smooth transformation.

Thanks a lot! Shan Chen

kennymckormick commented 2 weeks ago

Has replied via email.

shan23chen commented 1 week ago

All the tsv datasets are here: https://huggingface.co/datasets/WorldMedQA/V/tree/main

Thanks for integrating WorldMedQA-V into VLMEvalkKit!

kennymckormick commented 19 hours ago

@shan23chen PR #572 has been merged now.