search

Vchitect / VBench

[CVPR2024 Highlight] VBench - We Evaluate Video Generation
https://vchitect.github.io/VBench-project/
Apache License 2.0
374 stars 14 forks source link

single dimension running problem #8

Closed GFENGG closed 1 month ago

GFENGG commented 5 months ago

Hi, the path "prompts/prompts_per_dimension" contains only a part of evaluation dimensions. So, if i want to evaluation the dimension of "background_consistent", and there is no proper prompt text file (maybe background_consistent.txt) for me to generation videos. So, what can i do to perform evaluation on "background_consistent"?

GFENGG commented 5 months ago

And the "prompts/all_category.txt" is used to evaluation the model on different categories? But in "VBench_full_info.json", there seems no prompts of "prompts/all_category.txt"?

yinanhe commented 5 months ago

Hi, the path "prompts/prompts_per_dimension" contains only a part of evaluation dimensions. So, if i want to evaluation the dimension of "background_consistent", and there is no proper prompt text file (maybe background_consistent.txt) for me to generation videos. So, what can i do to perform evaluation on "background_consistent"?

@GFENGG Hi , you can refer to this file. On the left are the corresponding evaluation dimensions, and on the right is the prompt list of left dimension.

For example, if you want to test background_consistency, you only need to sample the video of scene.

https://github.com/Vchitect/VBench/blob/8046fcbfa1b853d6121956b9f48021de5f1fecb0/dimension_to_folder.json#L2-L17

yinanhe commented 5 months ago

And the "prompts/all_category.txt" is used to evaluation the model on different categories? But in "VBench_full_info.json", there seems no prompts of "prompts/all_category.txt"?

@GFENGG all_category is not a standalone dimension, so it will not appear in the json.

prompts/all_category.txt is a collection of video prompts for all dimensions that need to be tested. By sampling all prompts from prompts/all_category.txt into videos, VBench will automatically assign corresponding evaluation dimensions to each sampled video for evaluation and provide results.

GFENGG commented 5 months ago

And the "prompts/all_category.txt" is used to evaluation the model on different categories? But in "VBench_full_info.json", there seems no prompts of "prompts/all_category.txt"?

@GFENGG all_category is not a standalone dimension, so it will not appear in the json.

prompts/all_category.txt is a collection of video prompts for all dimensions that need to be tested. By sampling all prompts from prompts/all_category.txt into videos, VBench will automatically assign corresponding evaluation dimensions to each sampled video for evaluation and provide results.

Thanks for your reply. In which file (maybe a .json file like "VBench_full_info.json") can i find the relation between the prompts in "prompts/all_category.txt" and corresponding evaluation dimensions?

yinanhe commented 5 months ago

@GFENGG Sorry for the late reply. In all_category, there are approximately 100 prompts for each category. We conducted VBench evaluation for all prompts in subject consistency, background consistency, motion smoothness, dynamic degree, aestheic quality, image quality and overall consistency without further screening corresponding relationships.

ziqihuangg commented 1 month ago

Hi, we're closing this issue as it appears your questions have been addressed. However, feel free to open a new issue or reopen this one, if you have further questions or if anything else comes up related to this issue.