search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.4k stars 114 forks source link

cmmmu #10

Closed phellonchen closed 6 months ago

phellonchen commented 6 months ago

when using '--tasks cmmmu' I got some wrongs.

assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names.

kcz358 commented 6 months ago

Hi, when I tried lmms_eval --tasks list, I can see cmmmu on the list and this means cmmmu can be loaded successfully.

image

If you are encountering this issue, maybe you can try lmms_eval --tasks list --verbosity DEBUG and see what error causing the tasks can not be loaded.

phellonchen commented 6 months ago

@kcz358 Thanks for your reply. Another question: My server is unable to access the HF dataset directly. How can I use the local dataset downloaded from 'lmms-lab/CMMMU' to evaluate the model?

kcz358 commented 6 months ago

@phellonchen

I haven't tried this but based on my understanding of the code and huggingface load_dataset api.

You may want to change the dataset_path in the _default_template_cmmmu_yaml to your local dataset path.

Our code loads huggingface dataset using the load_dataset method. For each tasks, the code logic is like this dataset = load_dataset(<dataset_path>, name = <dataset_name> (if there is one else None for this), split = <split>). Thus, as long as you clone our dataset repo and can successfully load the dataset through load_dataset method, this should be able to work.