fudan-generative-vision / hallo

Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation
https://fudan-generative-vision.github.io/hallo/
MIT License
8.98k stars 1.21k forks source link

When FPS>25, OSError: Error in file #80

Open yincangshiwei opened 2 months ago

yincangshiwei commented 2 months ago

When FPS>25, OSError: Error in file appears. I tried 60FPS, 50FPS, and 30FPS, but this error occurred:

50FPS image image

60FPS image image

There is also a problem with 30FPS, if 25FPS is not a problem

subazinga commented 2 months ago

At present, only a frame rate of 25 FPS is supported due to technical constraints.

yincangshiwei commented 2 months ago

At present, only a frame rate of 25 FPS is supported due to technical constraints.由于技术限制,目前仅支持 25 FPS 的帧速率。

Okay, may I ask what other parameters besides FPS can be adjusted to maximize performance and make the video better.

xumingw commented 2 months ago

We optimize the model with current parameters; changing other parameters may degrade performance.

yincangshiwei commented 2 months ago

We optimize the model with current parameters; changing other parameters may degrade performance.

Can I use other SD1.5 models, especially the SD1.5 paint model? I see that your technique probably involves redrawing to make changes to the image. Can you try using some good redrawing models or methods? Currently, the best redrawing method is to use "Brushnet" technique.

xumingw commented 2 months ago

We optimize the model with current parameters; changing other parameters may degrade performance.

Can I use other SD1.5 models, especially the SD1.5 paint model? I see that your technique probably involves redrawing to make changes to the image. Can you try using some good redrawing models or methods? Currently, the best redrawing method is to use "Brushnet" technique.

Actually, we do not use any redrawing. Simply replacing the SD model won't work because we fine-tune some parameters. We are planning to train the model with a more powerful one in the near future.

Quest4AiJ commented 2 months ago

We are planning to train the model with a more powerful one in the near future.

@xumingw Is the newly released SD3 or its new VAE part under consideration to train a more powerful more?

Here is an interesting article about SD3 and its VAE: https://www.reddit.com/r/StableDiffusion/comments/1dcuval/the_importance_of_stable_diffusion_3_its_standout/