search

OpenGVLab / MMT-Bench

ICML'2024 | MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
72 stars 2 forks source link

dataset option issue #5

Open expapa opened 1 month ago

expapa commented 1 month ago

Thanks for your contribution, nice work.

{'index': 22147, 'question': 'Please write a poem based on the image.', 'answer': 'J', 'A': "Promise the joy that I've longed to hear", 'B': 'The essence of what you could be', 'C': 'Space you fill in my memory', 'D': 'Visions of our time apart', 'E': 'Promise a whisper that I need to hear', 'F': 'Profound impact, in my youth', 'G': "Promise the stars that I've aimed to hold", 'H': 'Illusions of the paths we walk', 'I': 'Thoughts of our future to come', 'category': 'writing_poetry_from_image', 'l2-category': 'visual_captioning', 'split': 'VAL'}

The entire dataset has no option of 'J', but the answer of the question in this sample is 'J', which is incorrect for researchers to use, any idea why?

KainingYing commented 4 weeks ago

Thanks for your attention on MMT-Bench.

Thank you for your careful reminder. This was a error in our dataset, which we have now corrected. Additionally, we have checked the other samples to ensure there are no further mistakes. The updated dataset will be released shortly.