search

Vchitect / VBench

[CVPR2024 Highlight] VBench - We Evaluate Video Generation
https://vchitect.github.io/VBench-project/
Apache License 2.0
575 stars 28 forks source link

Opensoraplan-v1.1 score #67

Closed LWang1016 closed 1 month ago

LWang1016 commented 2 months ago

@lmxyy @xjxu21 @piddnad @JustinYuu @simplaj , Hi, every, I just evaluate the Opensoraplan-v1.1 score with your provided videos and evaluation toolkit. But I can not get the same score as your report in the leadboard.
Besieds, I also evaluate the opensorav1.0, also with your provided videos and toolkit, and can get the same score.
I want to make sure for the Opensoraplan-v1.1 results, do you use the released videos for evaluation?

Thanks very much.

ziqihuangg commented 2 months ago

May I ask which evaluation pipeline did you use for evaluation? Is it VBench or VBench-Long? Thanks!

LWang1016 commented 2 months ago

VBench

LWang1016 commented 2 months ago

here is my evaluation results.

截屏2024-09-06 16 22 34
MqLeet commented 2 months ago

Hi @LWang1016 , I notice that you use VBench tools to evaluate Opensorav1-1, which will not get the results we published in the leaderboard.

We recommend you to use VBench-Long to evaluate Opensorav1-1, because this version generates long videos ~10s. You will get the same results in the leaderboard by using VBench-Long. Thanks!

Contact us if you have other questions.

LWang1016 commented 1 month ago

@MqLeet. I just download your released generated videos to do the metric evaluation.

MqLeet commented 1 month ago

Hi, @LWang1016, we will close this issue.

Feel free to contact us if you have other questions.