search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.02k stars 52 forks source link

How to contribute a new dataset? #99

Open JohnTang93 opened 3 weeks ago

JohnTang93 commented 3 weeks ago

We (ByteDance + Huazhong University of Science and Technology) have recently constructed a benchmark(MTVQA) to measure the multilingual text comprehension ability of LMMs, including 9 widely-used but low-resource languages, which demonstrates that there is still a large room for LMMs to improve the ability of multilingual text perception and comprehension. MTVQA bench wishes to elevate the multimodal research community's attention to a wider range of visual texts. Would it be possible to add MTVQA to your collection?
MTVQA: https://github.com/bytedance/MTVQA

kcz358 commented 2 weeks ago

Hi, you can refer to PR such as

95 #28 #100 #52

So basically you need to first prepare your dataset on huggingface and then prepare the task config in the tasks folder