search

hpcaitech / Open-Sora

Open-Sora: Democratizing Efficient Video Production for All
https://hpcaitech.github.io/Open-Sora/
Apache License 2.0
20.76k stars 1.97k forks source link

Training loss problem #402

Closed COST-97 closed 1 month ago

COST-97 commented 2 months ago

Hello: I used some new data to fine-tune on opensora1.1, but the noise loss can only drop from about 1 to about 0.2, and cannot continue to decrease. And after training for a long enough time, the loss increases instead. Did your model encounter similar problems during the training process?

qian18long commented 2 weeks ago

Hi, I have similar problem. How did you solve this problem?