search

Oryx-mllm / Oryx

MLLM for On-Demand Spatial-Temporal Understanding at Arbitrary Resolution
https://oryx-mllm.github.io
264 stars 11 forks source link

Evaluation on MVBench #14

Open yxsysu opened 17 hours ago

yxsysu commented 17 hours ago

Hi authors, thanks for your great work. I try to evaluate Oryx-7b on MVBench and use the script:

export HF_HOME="/mnt/data/datasets/hub"

export GPT_EVAL_VERSION="gpt-3.5-turbo" export OPENAI_API_KEY="xxxx" export OPENAI_API_URL="xxxx"

export LOWRES_RESIZE=384x32 export VIDEO_RESIZE="0x64" export HIGHRES_BASE="0x32" export MAXRES=1536 export MINRES=0 export VIDEO_MAXRES=480 export VIDEO_MINRES=288

MODEL_NAME="oryx_7b" MODEL_PATH="/mnt/data/Oryx-main/checkpoints/oryx_7b"

accelerate launch --num_processes=1 -m lmms_eval --model oryx --model_args pretrained=$MODEL_PATH,max_frames_num=64,mm_resampler_type="dynamic_compressor" --tasks mvbench --batch_size 1 --log_samples --log_samples_suffix eval --output_path ./logs_eval/$MODEL_NAME --verbosity DEBUG

Although the dataset is automatically downloaded, the program shut down. The log is: 10-16 22:06:51 [lmms_eval/api/task.py:801] INFO Extracted all files from /mnt/data/datasets/hub/hub/datasets--OpenGVLab--MVBench/snapshots/f6b751d0a8af80ea42331abacde0a481[638/1903] ideo/tvqa.zip to /mnt/data/datasets/hub/mvbench
10-16 22:06:53 [lmms_eval/api/task.py:801] INFO Extracted all files from /mnt/data/datasets/hub/hub/datasets--OpenGVLab--MVBench/snapshots/f6b751d0a8af80ea42331abacde0a481813eb5f1/v ideo/vlnqa.zip to /mnt/data/datasets/hub/mvbench
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/tenacity/init.py", line 478, in call
result = fn(*args, **kwargs)
File "/mnt/data/Oryx-main/lmms-eval/lmms_eval/api/task.py", line 845, in download
self.dataset = datasets.load_from_disk(os.path.join(self.DATASET_PATH, self.DATASET_NAME))
File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2630, in load_from_disk
raise FileNotFoundError(f"Directory {dataset_path} not found")
FileNotFoundError: Directory OpenGVLab/MVBench/action_antonym not found

I know the MVBench dataset contains the json files and video data, but does not contain a folder or a file named "MVBench/task_name"

Could you help me fix this bug? Looking forward to your reply. Thank you.

dongyh20 commented 10 hours ago

Looks like the folder do not contain a specific subtask of MVBench. For video evaluation using lmms-eval, it is recommended to download the zip files provided by the lmms-eval team, which contains raw videos of the benchmark. Then, you can set you HF_HOME to the folder.