search

ali-vilab / VGen

Official repo for VGen: a holistic video generation ecosystem for video generation building on diffusion models
https://i2vgen-xl.github.io
2.97k stars 265 forks source link

When can I expect to have the code of DreamVideo? Thanks. #67

Open lmm077 opened 10 months ago

lmm077 commented 10 months ago

DreamVideo

weilllllls commented 10 months ago

@lmm077 Hello, thank you for your attention to DreamVideo. Ideally, we plan to open source the code in about one to two months. We need to go through the internal approval process and are busy with other ongoing work. Please understand and be patient, thank you.

tyrink commented 8 months ago

@lmm077 Hello, thank you for your attention to DreamVideo. Ideally, we plan to open source the code in about one to two months. We need to go through the internal approval process and are busy with other ongoing work. Please understand and be patient, thank you.

Hi, two months have passed, and what is the progress of the open-source of dreamvideo?

weilllllls commented 8 months ago

@lmm077 Hello, thank you for your attention to DreamVideo. Ideally, we plan to open source the code in about one to two months. We need to go through the internal approval process and are busy with other ongoing work. Please understand and be patient, thank you.

Hi, two months have passed, and what is the progress of the open-source of dreamvideo?

Hello, we will open source the code before the end of this month. We have already cleaned up the code and are about to submit it for internal review.

weilllllls commented 7 months ago

Hi guys, the code is released.

XiaominLi1997 commented 7 months ago

Hi guys, the code is released.

Hello, thanks for releasing the code of Dreamvideo, but I am not sure about the following questions:

  1. Can I generate videos with any resolutions? like 512x512 or 576x320
  2. could you plz release more pretrained models?
  3. Given some training videos with motion "bow", the inference videos cannot produce correct motion, especially not aligned with the given prompt. (1) a panda is bowing under the tree (2) an alien is bowing on the Mars

https://github.com/ali-vilab/VGen/assets/25433111/8155eae8-e76e-4128-8ade-d40dbe5a84cd

https://github.com/ali-vilab/VGen/assets/25433111/42d92209-9c09-4111-adf2-9d38709116db

Thanks again for your nice work.

weilllllls commented 7 months ago

Hi guys, the code is released.

Hello, thanks for releasing the code of Dreamvideo, but I am not sure about the following questions:

  1. Can I generate videos with any resolutions? like 512x512 or 576x320
  2. could you plz release more pretrained models?

  3. Given some training videos with motion "bow", the inference videos cannot produce correct motion, especially not aligned with the given prompt. (1) a panda is bowing under the tree (2) an alien is bowing on the Mars

    A_panda_is_bowing_under_the_tree_8888_5.mp4 An_alien_is_bowing_on_Mars_8888_1.mp4 Thanks again for your nice work.

Hello, thanks for your interest.

  1. We recommend using the same resolution as the base model. In our experiments, the base model was trained with a resolution of 256x256.
  2. Yes, we will release models in a few days.
  3. (a) Have you put the reference images of panda and alien in the motionReferenceImgs folder and edited the config file? (b) What about using more training steps or a larger learning rate? (c) What about trying different seeds? If none of the above solves the problem, you can send the training videos and configuration file to my email "yjwei22@m.fudan.edu.cn" and I will gladly work this out.
XiaominLi1997 commented 7 months ago

Hi guys, the code is released.

Hello, thanks for releasing the code of Dreamvideo, but I am not sure about the following questions:

  1. Can I generate videos with any resolutions? like 512x512 or 576x320
  2. could you plz release more pretrained models?
  3. Given some training videos with motion "bow", the inference videos cannot produce correct motion, especially not aligned with the given prompt. (1) a panda is bowing under the tree (2) an alien is bowing on the Mars

A_panda_is_bowing_under_the_tree_8888_5.mp4 An_alien_is_bowing_on_Mars_8888_1.mp4 Thanks again for your nice work.

Hello, thanks for your interest.

  1. We recommend using the same resolution as the base model. In our experiments, the base model was trained with a resolution of 256x256.
  2. Yes, we will release models in a few days.
  3. (a) Have you put the reference images of panda and alien in the motionReferenceImgs folder and edited the config file? (b) What about using more training steps or a larger learning rate? (c) What about trying different seeds? If none of the above solves the problem, you can send the training videos and configuration file to my email "yjwei22@m.fudan.edu.cn" and I will gladly work this out.

Thank you for replying so quickly.

  1. I am not sure what "the base model" refers to. Does it denote modelscope or sd2.1?
  2. Waiting for your nice models.
  3. (a) I did not use reference images, during inference, I set "use_clip_adapter_condition" to False. Just use motion customization without subject customization. (b) Yep. I need to use more training steps, I just train the model for only 600 steps before. (c) changing seeds may help a lot. Thx again.
weilllllls commented 7 months ago

Hi guys, the code is released.

Hello, thanks for releasing the code of Dreamvideo, but I am not sure about the following questions:

  1. Can I generate videos with any resolutions? like 512x512 or 576x320
  2. could you plz release more pretrained models?
  3. Given some training videos with motion "bow", the inference videos cannot produce correct motion, especially not aligned with the given prompt. (1) a panda is bowing under the tree (2) an alien is bowing on the Mars

A_panda_is_bowing_under_the_tree_8888_5.mp4 An_alien_is_bowing_on_Mars_8888_1.mp4 Thanks again for your nice work.

Hello, thanks for your interest.

  1. We recommend using the same resolution as the base model. In our experiments, the base model was trained with a resolution of 256x256.
  2. Yes, we will release models in a few days.
  3. (a) Have you put the reference images of panda and alien in the motionReferenceImgs folder and edited the config file? (b) What about using more training steps or a larger learning rate? (c) What about trying different seeds? If none of the above solves the problem, you can send the training videos and configuration file to my email "yjwei22@m.fudan.edu.cn" and I will gladly work this out.

Thank you for replying so quickly.

  1. I am not sure what "the base model" refers to. Does it denote modelscope or sd2.1?
  2. Waiting for your nice models.
  3. (a) I did not use reference images, during inference, I set "use_clip_adapter_condition" to False. Just use motion customization without subject customization. (b) Yep. I need to use more training steps, I just train the model for only 600 steps before. (c) changing seeds may help a lot. Thx again.

The base model refers to the video diffusion model (ModelScopeT2V) and we do not use SD. Thanks for your patience.

weilllllls commented 7 months ago

@XiaominLi1997 We have released the model weights and provided some examples, please refer to the latest code.