how to get three pope results [rad,pop,adv] with lmms-eval?

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval

https://lmms-lab.github.io/

Other

1.39k stars 110 forks source link

how to get three pope results [rad,pop,adv] with lmms-eval? #85

Closed baiyuting closed 4 months ago

baiyuting commented 4 months ago

I find the pope results reported in the paper are like this.

how to use lmms-eval to get such three results? I only get results as below, there is no such metric.

kcz358 commented 4 months ago

Hi, thank you for the issue. I have created new tasks group to output the score separately. You can pull the main branch again and run with --tasks pope_full. Here are my results using llava_hf llava-1.5-7b.

Feel free to check whether my implementation is correct or not

baiyuting commented 4 months ago

I run command and find it work well. By now I think it is ok. thanks