search

MMStar-Benchmark / MMStar

This repo contains evaluation code for the paper "Are We on the Right Way for Evaluating Large Vision-Language Models"
https://mmstar-benchmark.github.io
131 stars 2 forks source link

Error when inference #2

Closed Cuiunbo closed 4 months ago

Cuiunbo commented 4 months ago

Here is output. prompt: Question: What is the main feature in the background of the image? Options: A: A park bench near the water., B: A couple sitting on a bench., C: A body of water and the Golden Gate Bridge., D: A mountain in the distance. 0%| | 0/188 [00:00<?, ?it/s] Traceback (most recent call last): File "/home/cuijunbo/eval_mm/MMStar/eval/run.py", line 89, in main() File "/home/cuijunbo/eval_mm/MMStar/eval/run.py", line 80, in main model = infer_data_job(model, work_dir=pred_root, model_name=model_name, dataset_name=dataset_name, File "/home/cuijunbo/eval_mm/MMStar/eval/vlmeval/inference.py", line 172, in infer_data_job model = infer_data(model, work_dir=work_dir, dataset_name=dataset_name, File "/home/cuijunbo/eval_mm/MMStar/eval/vlmeval/inference.py", line 113, in infer_data struct = dataset.build_prompt(data.iloc[i], for_llm=is_llm) File "/home/cuijunbo/eval_mm/MMStar/eval/vlmeval/utils/dataset.py", line 77, in build_prompt tgt_path = self.dump_image(line, dataset) File "/home/cuijunbo/eval_mm/MMStar/eval/vlmeval/utils/custom_prompt.py", line 33, in dump_image decode_base64_to_image_file(line['image'], tgt_path) File "/home/cuijunbo/eval_mm/MMStar/eval/vlmeval/smp/vlm.py", line 73, in decode_base64_to_image_file image = decode_base64_to_image(base64_string, target_size=target_size) File "/home/cuijunbo/eval_mm/MMStar/eval/vlmeval/smp/vlm.py", line 63, in decode_base64_to_image image_data = base64.b64decode(base64_string) File "/home/cuijunbo/miniconda3/envs/minicpm-v/lib/python3.10/base64.py", line 87, in b64decode return binascii.a2b_base64(s) binascii.Error: Incorrect padding

xiaoachen98 commented 4 months ago

Here is output. prompt: Question: What is the main feature in the background of the image? Options: A: A park bench near the water., B: A couple sitting on a bench., C: A body of water and the Golden Gate Bridge., D: A mountain in the distance. 0%| | 0/188 [00:00<?, ?it/s] Traceback (most recent call last): File "/home/cuijunbo/eval_mm/MMStar/eval/run.py", line 89, in main() File "/home/cuijunbo/eval_mm/MMStar/eval/run.py", line 80, in main model = infer_data_job(model, work_dir=pred_root, model_name=model_name, dataset_name=dataset_name, File "/home/cuijunbo/eval_mm/MMStar/eval/vlmeval/inference.py", line 172, in infer_data_job model = infer_data(model, work_dir=work_dir, dataset_name=dataset_name, File "/home/cuijunbo/eval_mm/MMStar/eval/vlmeval/inference.py", line 113, in infer_data struct = dataset.build_prompt(data.iloc[i], for_llm=is_llm) File "/home/cuijunbo/eval_mm/MMStar/eval/vlmeval/utils/dataset.py", line 77, in build_prompt tgt_path = self.dump_image(line, dataset) File "/home/cuijunbo/eval_mm/MMStar/eval/vlmeval/utils/custom_prompt.py", line 33, in dump_image decode_base64_to_image_file(line['image'], tgt_path) File "/home/cuijunbo/eval_mm/MMStar/eval/vlmeval/smp/vlm.py", line 73, in decode_base64_to_image_file image = decode_base64_to_image(base64_string, target_size=target_size) File "/home/cuijunbo/eval_mm/MMStar/eval/vlmeval/smp/vlm.py", line 63, in decode_base64_to_image image_data = base64.b64decode(base64_string) File "/home/cuijunbo/miniconda3/envs/minicpm-v/lib/python3.10/base64.py", line 87, in b64decode return binascii.a2b_base64(s) binascii.Error: Incorrect padding

Sorry for our mistake. This is due to the damaged 'tsv' file. We have already fixed this issue. You can clean the cached 'MMStar.tsv' file and 'images/MMStar' image files located in your '~/LMUData' directory.

All you need to do is clear the cache file and run the command in eval/scripts/bash.sh and the download of the new full file will take place automatically.