search

PKU-YuanGroup / Open-Sora-Plan

This project aim to reproduce Sora (Open AI T2V model), we wish the open source community contribute to this project.
MIT License
11.25k stars 1.01k forks source link

Train with Ckpts with Different Frames #433

Open GloryyrolG opened 1 week ago

GloryyrolG commented 1 week ago

Hi @LinB203 @qqingzheng @clownrat6 et al.,

may i ask What we should pay special attention to when resuming training from a checkpoint with a different temporal length, e.g. train 93 with a 29 ckpt (stage 5) or vice versa? The 93-frame model does not generate good 29-frame videos due to the use of positional embeddings. thx!

GloryyrolG commented 1 week ago

E.g., i found For training 29x480p_i2v from 93x480p_i2v with a stronger prior, the reconstruction capability can be adapted in just a few steps; and after about 100 steps, a certain degree of temporal video generation can be restored.