search

Vchitect / VBench

[CVPR2024 Highlight] VBench - We Evaluate Video Generation
https://vchitect.github.io/VBench-project/
Apache License 2.0
374 stars 14 forks source link

Evaluation data issues #12

Closed GFENGG closed 2 months ago

GFENGG commented 5 months ago

Hello, Is there an unified configs for the evaluation data (mp4 files), such as the fps, duration, resolution, and etc.

And does the different settings (such as duration and resolution) of mp4 files have an influence on the final evaluation results using VBench? I'm not sure about this.

ziqihuangg commented 4 months ago

Hi, thanks for your question! Currently there are several supported formats (like MP4 and GIF). See this function for video loading: https://github.com/Vchitect/VBench/blob/master/vbench/utils.py#L101

VBench has already considered the influence from frame rate and resolution during evaluation.

For example, some dimensions can be sensitive to video frame rate, like "Dynamic Degree". For higher frame rate videos, the dynamics (movement) between two consecutive frames are generally smaller, resulting in a false result of "low dynamic degree". Therefore, we eliminate frame rate's influence at the preprocessing step: https://github.com/Vchitect/VBench/blob/master/vbench/dynamic_degree.py#L86C9-L86C19

So during evaluation, you can directly supply the raw video without any preprocessing to VBench, and VBench will take care of any preprocessing needed for individual evaluation dimensions.