search

open-compass / VLMEvalKit

Open-source evaluation toolkit of large vision-language models (LVLMs), support 160+ VLMs, 50+ benchmarks
https://huggingface.co/spaces/opencompass/open_vlm_leaderboard
Apache License 2.0
1.39k stars 194 forks source link

MMBench-Video Dataset Download Bug #589

Closed Noctis-SC closed 1 week ago

Noctis-SC commented 1 week ago

Hello, I have an issue with executing the evaluation with MMBench-Video dataset:

I try to execute run.py with MMBench-Video dataset, but there are two scenarios that happen everytime I try to run the program. It either says that repo path is not a directory or downloads the dataset again (even though I have downloaded it already).

Here is my huggingface cache directory: (evalkit) root@autodl-container-3fac46b1d9-b311c924:~/autodl-tmp/hf_cache/huggingface/hub# pwd /root/autodl-tmp/hf_cache/huggingface/hub

(evalkit) root@autodl-container-3fac46b1d9-b311c924:~/autodl-tmp/hf_cache/huggingface/hub# ls MVLU datasets--opencompass--MMBench-Video models--Qwen--Qwen2-VL-2B-Instruct datasets--MLVU--MVLU models--Efficient-Large-Model--VILA1.5-3b version.txt

When I export the path to the huggingface cache directory (without the hub included): (evalkit) root@autodl-container-3fac46b1d9-b311c924:~/autodl-tmp/sanya/VLMEvalKit# export HF_HOME=/root/autodl-tmp/hf_cache/huggingface and execute: python run.py --data MMBench-Video --model GPT4o

Then I receive this error: raise CorruptedCacheException(f"Repo path is not a directory: {repo_path}") huggingface_hub.errors.CorruptedCacheException: Repo path is not a directory: /root/autodl-tmp/hf_cache/huggingface/datasets--opencompass--MMBench-Video

When I export the path to the huggingface cache directory (the hub included): export HF_HOME=/root/autodl-tmp/hf_cache/huggingface/hub and execute: Then it starts to download a dataset under the following path: /root/autodl-tmp/hf_cache/huggingface/hub/hub

FangXinyu-0913 commented 1 week ago

Hi @Noctis-SC. Thanks for your advice. I have fixed the problem, you can pull the latest code and try again, if you still have problems with re-downloading： Then it starts to download a dataset under the following path: Before the redownload action begins, it will check the integrity of this function(https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/mmbench_video.py#L70), and it will turn to false if it can not find any of the videos or the md5 code isn't the same as setting. So could you please debug the code and tell me it turn to false in which part of this function.

Noctis-SC commented 1 week ago

Hi @Noctis-SC. Thanks for your advice. I have fixed the problem, you can pull the latest code and try again, if you still have problems with re-downloading： Then it starts to download a dataset under the following path: Before the redownload action begins, it will check the integrity of this function(https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/mmbench_video.py#L70), and it will turn to false if it can not find any of the videos or the md5 code isn't the same as setting. So could you please debug the code and tell me it turn to false in which part of this function.

Thank you for your quick response, @FangXinyu-0913 I have pulled the updated code, it seems to work fine with VILA1.5-3Bmodel. However, I'm not sure is it related or not, but I have another issue, this time with Qwen2-VL-7B-Instruct

When I run this command: (qwen2) root@autodl-container-3fac46b1d9-b311c924:~/autodl-tmp/sanya/VLMEvalKit# AUTO_SPLIT=1 python run.py --model Qwen2-VL-7B-Instruct --data MMBench-Video --verbose

The output I get: /root/autodl-tmp/miniconda3/envs/qwen2/lib/python3.10/site-packages/transformers/utils/hub.py:128: FutureWarning: UsingTRANSFORMERS_CACHEis deprecated and will be removed in v5 of Transformers. UseHF_HOMEinstead. warnings.warn( [2024-11-11 14:26:33,312] WARNING - RUN - run.py: main - 53: --reuse is not set, will start the evaluation from scratch [2024-11-11 14:26:33] WARNING - run.py: main - 53: --reuse is not set, will start the evaluation from scratch Downloading shards: 100%|████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 18542.46it/s] Qwen2VLRotaryEmbeddingcan now be fully parameterized by passing the model config through theconfigargument. All other arguments will be removed in v4.46 Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████| 5/5 [00:05<00:00, 1.11s/it] 0it [00:00, ?it/s]Qwen2-VL-7B-Instruct is a video-llm model, nframe is set to 8, not using default [{'role': 'user', 'content': [{'type': 'text', 'text': 'What is the name of the player who scored the first goal in the video?'}, {'type': 'video', 'video': 'file:///root/autodl-tmp/hf_cache/huggingface/datasets/hub/datasets--opencompass--MMBench-Video/snapshots/ef35e21df54488715a906c7e47146f5d9f4abbed/video/wZxzBvAgqxc.mp4', 'nframes': 8}]}] qwen-vl-utils using decord to read video. The player who scored the first goal in the video is wearing a red jersey with the name "PALMER" on the back. The player who scored the first goal in the video is wearing a red jersey with the name "PALMER" on the back. 1it [00:04, 4.07s/it] [2024-11-11 14:26:46,632] ERROR - RUN - run.py: main - 284: Model Qwen2-VL-7B-Instruct x Dataset MMBench-Video combination failed: '>' not supported between instances of 'NoneType' and 'int', skipping this combination. Traceback (most recent call last): File "/root/autodl-tmp/sanya/VLMEvalKit/run.py", line 163, in main model = infer_data_job_video( File "/root/autodl-tmp/sanya/VLMEvalKit/vlmeval/inference_video.py", line 173, in infer_data_job_video model = infer_data( File "/root/autodl-tmp/sanya/VLMEvalKit/vlmeval/inference_video.py", line 98, in infer_data if getattr(model, 'fps', 0) > 0: TypeError: '>' not supported between instances of 'NoneType' and 'int' [2024-11-11 14:26:46] ERROR - run.py: main - 284: Model Qwen2-VL-7B-Instruct x Dataset MMBench-Video combination failed: '>' not supported between instances of 'NoneType' and 'int', skipping this combination. Traceback (most recent call last): File "/root/autodl-tmp/sanya/VLMEvalKit/run.py", line 163, in main model = infer_data_job_video( File "/root/autodl-tmp/sanya/VLMEvalKit/vlmeval/inference_video.py", line 173, in infer_data_job_video model = infer_data( File "/root/autodl-tmp/sanya/VLMEvalKit/vlmeval/inference_video.py", line 98, in infer_data if getattr(model, 'fps', 0) > 0: TypeError: '>' not supported between instances of 'NoneType' and 'int'

FangXinyu-0913 commented 1 week ago

Hi @Noctis-SC. I have already fixed this bug in the newest code. This is due to problems with fps and nframe Settings. You can pull the latest code and execute the same command, and there are no more problems in my attempts.

Noctis-SC commented 1 week ago

Hello, @FangXinyu-0913, thank you for your quick fix, everything works for me now. Thank you again