search

baaivision / Emu

Emu Series: Generative Multimodal Models from BAAI
https://baaivision.github.io/emu2/
Apache License 2.0
1.66k stars 86 forks source link

Invitation to support EMU2 in VLMEvalKit #64

Open kennymckormick opened 10 months ago

kennymckormick commented 10 months ago

Dear Authors, Congratulates to this great work. I'm Kenny from OpenCompass and we are building a comprehensive benchmark for multi-modality understanding. We would like to invite you to implement your great work -- EMU2 in our multi-modal evaluation benchmark VLMEvalKit. By implementing your work in this framework, you can easily utilize our 1-click solution to evaluation your VLM on tens of different benchmarks. Besides, according to your wishes, we can help include your model into the MMBench leaderboard and OpenCompass-MM leaderboard.

kennymckormick commented 10 months ago

@Quan-Sun @yqy2001 Dear authors, will you consider this proposition? Actually we are glad to conduct this on our own, but we may be not sure that if the default prompts can reproduce your reported results.

yqy2001 commented 10 months ago

Dear Kenny,

Thank you for your kind invitation! We are willing to add Emu2 to the VLMEvalKit multimodal benchmark and leaderboard.

Please establish contact with xszhang@baai.ac.cn and we will provide the prompt details for evaluation. We appreciate your help and effort :).