search

MMMU-Benchmark / MMMU

This repo contains evaluation code for the paper "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI"
https://mmmu-benchmark.github.io/
Apache License 2.0
353 stars 24 forks source link

Error reports when loading the dataset #4

Closed XiongweiWu closed 11 months ago

XiongweiWu commented 11 months ago

Hi, I am try to load your dataset with the provided command: from datasets import load_dataset dataset = load_dataset("MMMU/MMMU")

However, one error reports as:

ExpectedMoreSplits: {'dev'}

Can u check it?

NipElement commented 11 months ago

Thank you for using our dataset! Because the dataset includes multiple subsets, and you need to specify which subset you want to load. To resolve this, you should specify the subset while loading the dataset like follow:

from datasets import load_dataset
dataset = load_dataset("MMMU/MMMU", 'Computer_Science')

Let us know if you have any further issues.

Yushi-Hu commented 11 months ago

I was meeting the same problem and I solved it by updating my datasets package. @XiongweiWu you may try it.

XiongweiWu commented 11 months ago

@Yushi-Hu Thx for your help! After upgrading the datasets to 2.15.0, the problem is solved.

drogozhang commented 11 months ago

Thanks @Yushi-Hu for the answer! Close this issue now for inactivity, feel free to re-open it if you have further questions.