search

Vchitect / VBench

[CVPR2024 Highlight] VBench - We Evaluate Video Generation
https://vchitect.github.io/VBench-project/
Apache License 2.0
482 stars 23 forks source link

Duplicate prompts in overall_consistency.txt and full.json #58

Open lovelyczli opened 1 month ago

lovelyczli commented 1 month ago

Thank you for the great work!

I noticed that there is a duplicate prompt "A fantasy landscape" in the prompts/prompts_per_dimension/overall_consistency.txt file on line 31 and line 52. A similar issue appears in the full.json file as well.

Additionally, we found only 92*5=460 videos in the provided Sampled Videos of the overall_consistency dimension, which seems to align with the duplicated prompt.

Although the differences in the results are minimal, VBench is a widely followed benchmark, and it would be great if this issue could be addressed soon.

Thanks again for your excellent work!

lovelyczli commented 1 month ago

@ziqihuangg

ziqihuangg commented 1 month ago

You are correct that there are two similar prompts in the prompts/prompts_per_dimension/overall_consistency.txt file, differing only in upper/lower case. Due to our current video naming protocol, the later prompt’s video overwrites the earlier one during sampling. This process is consistently applied across all models under evaluation to ensure fair comparison. Thus, while it doesn’t affect fairness, it does result in extra time spent sampling one additional prompt out of the 900+ prompts. We opted not to alter the prompt list to ensure that all results remain fully reproducible. Hope this clarifies, thanks!

lovelyczli commented 1 month ago

I got it, thank you for the quick response!