search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.03k stars 54 forks source link

llava_bench_wild seems not a valid parameter #2

Closed wusize closed 3 months ago

wusize commented 3 months ago

First of all, a big thanks to all members of this open-source project, which definitely shortens the development cycle of LMMs for the whole community.

I tested my model on a series of benchmarks (e.g., seedbench, mmvet,ai2d, chartqa, llava_bench_wild, docvqa_val) and found llava_bench_wild did not appear in the result jsons. When I passed only llava_bench_wild in the arguments, I got the following error message assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names.".

kcz358 commented 3 months ago

Hi, sorry for the inconvenience.

The task name for llava-in-the-wild in the readme is wrong and you might want to try using tasks=llava_in_the_wild.

If you want to check all the possible tasks, you can use lmms_eval --tasks list to check. Noted that this would print out all the tasks and groups (consist of several tasks) that can be run for test.