search

hpcaitech / Open-Sora

Open-Sora: Democratizing Efficient Video Production for All
https://hpcaitech.github.io/Open-Sora/
Apache License 2.0
21.76k stars 2.1k forks source link

请问1.2的生成结果为花屏是什么原因 #477

Closed dengdeng-cat closed 3 months ago

dengdeng-cat commented 3 months ago

! 20240619-202254 请问1.2的生成结果为花屏是什么原因，是哪一个库版本不对么？ python scripts/inference.py configs/opensora-v1-2/inference/sample.py \ --num-frames 2s --resolution 360p \ --layernorm-kernel False --flash-attn False \ --prompt "a beautiful waterfall"

zhengzangw commented 3 months ago

我尝试你了你命令是可以正常生成的。花屏基本是由于权重没有正确加载导致的，我觉得有两个可能：

  1. 命令如果换行，需要加 \

    python scripts/inference.py configs/opensora-v1-2/inference/sample.py \
--num-frames 2s --resolution 360p \
--layernorm-kernel False --flash-attn False \
--prompt "a beautiful waterfall"

  2. 请问你有修改 config 吗？因为我们的权重应该都会自动下载。你运行的过程中有看到权重在下载的进度条吗？

dengdeng-cat commented 3 months ago

我尝试你了你命令是可以正常生成的。花屏基本是由于权重没有正确加载导致的，我觉得有两个可能：

  1. 命令如果换行，需要加 \
python scripts/inference.py configs/opensora-v1-2/inference/sample.py \
--num-frames 2s --resolution 360p \
--layernorm-kernel False --flash-attn False \
--prompt "a beautiful waterfall"
  1. 请问你有修改 config 吗？因为我们的权重应该都会自动下载。你运行的过程中有看到权重在下载的进度条吗？

非常感谢您的回复！

  1. 我实际运行时加了“\”的，这个原因应该不成立；
  2. config我只是把和apex和flash-attn的两个开关关闭了，然后我是提前下载好模型文件，按照config中原始的路径放好从本地加载的，不过好像运行的时候这两个新的模型并没有显示加载的进度条，如图所示： img_v3_02c1_1e4f5919-5cfb-4bd6-b302-b00c6138019g
dengdeng-cat commented 3 months ago

对了，补充一点，我还将配置文件中的“df16”改成了“fp16”，因为我的机器不支持，这一点在1.1中验证是没有问题的。

catsled commented 3 months ago

对了，补充一点，我还将配置文件中的“df16”改成了“fp16”，因为我的机器不支持，这一点在1.1中验证是没有问题的。

同样的问题，我使用的也是fp16，怀疑是fp16的问题 image

zhengzangw commented 3 months ago

We confirm this is a bug and is working on it.

zhengzangw commented 3 months ago

The problem is fixed by PR #523

dengdeng-cat commented 3 months ago

The problem is fixed by PR #523

Thanks.

CIntellifusion commented 2 months ago

I got a similar collapsed video using sample.sh from stand vbench prompts. I checked

  1. the ckpt path, both hpcai/STDiT and my fine-tuned checkpoint by printing it during initialization
  2. using gradio/app.py to generate video with two ckpt which is all right. Do you have any ideas about this? I think the backbone of load ckpt are load_pretrained for gradio/app.py and inference.py
CIntellifusion commented 2 months ago

I got a similar collapsed video using sample.sh from stand vbench prompts. I checked

  1. the ckpt path, both hpcai/STDiT and my fine-tuned checkpoint by printing it during initialization
  2. using gradio/app.py to generate video with two ckpt which is all right. Do you have any ideas about this? I think the backbone of load ckpt are load_pretrained for gradio/app.py and inference.py

Sorry, this issue has been solved. That is because the num_frame in sample.sh should be passed 2s instead of 16.

ChengYoghurt commented 1 month ago

Sorry, this issue has been solved. That is because the num_frame in sample.sh should be passed 2s instead of 16.

Hello, I'm also trying to use sample.sh provided by Open-Sora. Yet I'm having trouble in getting the ckpt path right.

I'm not sure if I need to download the pre-trained weights from huggingface beforehand.

Would you mind sharing your thoughts on it?

It would help a lot. Thanks in advance!