Supporting Multimodality

EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.

https://www.eleuther.ai

MIT License

5.8k stars 1.54k forks source link

Supporting Multimodality #2014

Open lintangsutawika opened 1 week ago

lintangsutawika commented 1 week ago

Add support to multimodal models as discussed with @haileyschoelkopf

This PR #1832 would be a great starting point.
List all tasks we want to support for the first iteration.

ashvinnihalani commented 4 days ago

Do we have an ETA for this? Happy to help contribute in any way I can. For reference we are doing part of our VLM evaluation suite in lm-evaluation-harness and some of it in some other frameworks and are looking to unify it.