search

open-compass / VLMEvalKit

Open-source evaluation toolkit of large vision-language models (LVLMs), support 160+ VLMs, 50+ benchmarks
https://huggingface.co/spaces/opencompass/open_vlm_leaderboard
Apache License 2.0
1.45k stars 198 forks source link

[Benchmark] Added MMGenBench benchmark #614

Closed lerogo closed 1 week ago

lerogo commented 1 week ago

[Benchmark] Added MMGenBench benchmark

lerogo commented 1 week ago

hey @kennymckormick do you think it's ready to merge?

kennymckormick commented 1 week ago

hey @kennymckormick do you think it's ready to merge?

Please help fix the lint first

lerogo commented 1 week ago

hey @kennymckormick do you think it's ready to merge?

Please help fix the lint first

It's done

kennymckormick commented 1 week ago

Hi, @lerogo , I didn't find any documents / descriptions about the MMGenBench you contributed on the Internet. Please add some introductions to the benchmark somewhere. If the benchmark is just for your personal use, you can also keep it in your own fork (no need to create a PR).

lerogo commented 1 week ago

Hi, @lerogo , I didn't find any documents / descriptions about the MMGenBench you contributed on the Internet. Please add some introductions to the benchmark somewhere. If the benchmark is just for your personal use, you can also keep it in your own fork (no need to create a PR).

We are sorry, we are working as quickly as possible to perfect our code and documentation. Below is the link to our paper: https://arxiv.org/abs/2411.14062

kennymckormick commented 1 week ago

Hi, @lerogo , I didn't find any documents / descriptions about the MMGenBench you contributed on the Internet. Please add some introductions to the benchmark somewhere. If the benchmark is just for your personal use, you can also keep it in your own fork (no need to create a PR).

We are sorry, we are working as quickly as possible to perfect our code and documentation. Below is the link to our paper: https://arxiv.org/abs/2411.14062

Cool, will try to run the evaluation and then merge it.

lerogo commented 1 week ago

Thanks! I've uploaded the code. Is it possible to add a link to our benchmark inside the VLMEvalKit homepage?

https://mmgenbench.alsoai.com/

kennymckormick commented 1 week ago

@lerogo Would you please also incorporate the evaluation part to deliver a more complete pipeline?

kennymckormick commented 1 week ago

Thanks! I've uploaded the code. Is it possible to add a link to our benchmark inside the VLMEvalKit homepage?

https://mmgenbench.alsoai.com/

Added a new line in the news section.

lerogo commented 1 week ago

Thanks! I've uploaded the code. Is it possible to add a link to our benchmark inside the VLMEvalKit homepage? https://mmgenbench.alsoai.com/

Added a new line in the news section.

Thanks! Can you please add it to Supported Benchmarks as well?

MMGenBench https://mmgenbench.alsoai.com/ MMGenBench-Test MMGenBench-Domain -

lerogo commented 1 week ago

dd