search

hpcaitech / Open-Sora

Open-Sora: Democratizing Efficient Video Production for All
https://hpcaitech.github.io/Open-Sora/
Apache License 2.0
21.76k stars 2.1k forks source link

The effect of generating human bodies is vastly different from what's shown in the Readme. #509

Closed Winn1y closed 2 months ago

Winn1y commented 3 months ago

https://github.com/hpcaitech/Open-Sora/assets/115919287/be3bc504-4242-42c4-9d21-cbc934b69de7

https://github.com/hpcaitech/Open-Sora/assets/115919287/4ceef340-ccdf-49f5-b2b4-6ea36ca70c87

so bad...

command:

python scripts/inference.py configs/opensora-v1-2/inference/sample.py --num-frames 8s --resolution 720p --aspect-ratio 1:1 --prompt "a woman shows her hands" "A father with a beard and glasses, wearing casual jeans and a t-shirt, pushes his laughing daughter on a swing in a sunny park. His face beams with happiness and love." --num-sampling-steps 100 --aes 8 --loop 2 --condition-frame-length 10

zhengzangw commented 3 months ago

Consistently generating a stable human body is quite a hard task. We admit that there are many failure cases for our current model, and we cherry-pick some good results as our demos. However, the following tricks may help improve the results:

  1. Use the --llm-refine with gpt4o refinement
  2. --aes 7 as the training data rarely has 8
  3. --flow 5 to reduce motion strength
  4. Also, you can try generating an image with our model first, and then generate the video with image2video.
jdietzChina commented 3 months ago

@zhengzangw This is amazing! Just got it running and am so happy to have any kind of video creation running. Thanks for the hard work!

github-actions[bot] commented 3 months ago

This issue is stale because it has been open for 7 days with no activity.