[rank0]: KeyError: 'answer'

[lucasjinreal]

/vlmeval/dataset/image_mcq.py", line 181, in evaluate rank0: answer_map = {i: c for i, c in zip(meta['index'], meta['answer'])} rank0: File "/data/miniconda3/envs/env-3.9.2/lib/python3.9/site-packages/pandas/core/frame.py", line 4102, in getitem rank0: indexer = self.columns.get_loc(key) rank0: File "/data/miniconda3/envs/env-3.9.2/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc rank0: raise KeyError(key) from err

[junming-yang]

Can you provide the running command?

[junming-yang]

If you are evaluating a private benchmark, please check your dataset file xxx. tsv fields correct, such as 'answer' instead of 'ans'. Please refer to Development Doc for more detailed information.

[lucasjinreal]

Am runing on MME only got this error

[junming-yang]

I try to reproduce this error. This is my command torchrun --nproc-per-node=8 run.py --model InternVL-Chat-V1-5 --data MME. Everything is OK. And MME benchmark should correspond to the image_yorn.py file instead of image_mcq.py. Please check your code repo.

[lucasjinreal]

Then that's weired, I didn touched anything inside vlmkit, those tsv are automatically downloaded, there must be an key error at some where in some dataset.

[junming-yang]

Maybe you can try to pull our latest code. We have fixed some known minor bugs. If you still encounter other problems, please do not hesitate to contact us and give us some detailed information to fix them. Thank you.

[kennymckormick]

Hi, @lucasjinreal , I tried to reproduce the problem but failed (I'm evaluating llava_v1.5_7b on MME), with latest main branch.