search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.39k stars 111 forks source link

when we can expect this support for more datasets to be completed?（Raven/NLVR2/IconQA） #40

Open littleSunlxy opened 6 months ago

littleSunlxy commented 6 months ago

Thank you for your wonderful works. We are looking forward to supporting more datasets. Could you please inform us when we can expect this support to be completed?

  1. Winoground (winoground)
  2. NLVR2 (nlvr2)
  3. RavenIQ-Test (raveniq)
  4. IconQA (iconqa)

Waiting for reply

Luodian commented 6 months ago

We do not have a clear timeline for it, and we are welcome the community support~

kcz358 commented 5 months ago

Hi @littleSunlxy , for iconqa, it is already supported in our pipeline

https://github.com/EvolvingLMMs-Lab/lmms-eval/tree/main/lmms_eval/tasks/iconqa