kennymckormick
commented
10 months ago Hi, @Isaachhh , the new version data file only works with our evaluation toolkit VLMEvalKit. It saves only one copy of image for different circular passes to reduce the file size. We strongly recommend you to use VLMEvalKit for evaluation (it supports evaluate llava-v1.5 on tens of benchmarks! )
By the way, I have updated the README file, if you would like to use the old version of file, you can download the legacy version.
Thanks for your wonderful work!
The MMBench-Test file has been updated in the latest commit.
When I used the evaluation script supplied by LLaVA-v1.5 to evaluate LLaVA-v1.5 on the latest
MMBench_TEST_EN.tsv, it occured an error:27%|███████████████ | 1784/6666 [05:04<13:52, 5.87it/s] Traceback (most recent call last): File "/root/anaconda3/envs/run/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/root/anaconda3/envs/run/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/LLaVA/llava/eval/model_vqa_mmbench.py", line 170, in
exec(code, run_globals)
File "/LLaVA/llava/eval/model_vqa_mmbench.py", line 170, in
eval_model(args)
File "/LLaVA/llava/eval/model_vqa_mmbench.py", line 84, in eval_model
image = load_image_from_base64(row['image'])
File "/LLaVA/llava/mm_utils.py", line 11, in load_image_from_base64 return Image.open(BytesIO(base64.b64decode(image))) File "/root/anaconda3/envs/run/lib/python3.10/base64.py", line 87, in b64decode return binascii.a2b_base64(s) binascii.Error: Incorrect padding
However, the code works well on the previous
mmbench_test_en_20231003.tsv. And I notice that the file size has reduced to 1/3. It may be some encoding-decoding bugs here.I use Python 3.10.
I would sincerely appreciate it if you figure out this problem because I didn't use the official VLMEvalKit. However, it's realy convenient to use the evaluation script included in the LLaVA repository.
Thanks a lot!