search

PKU-YuanGroup / Open-Sora-Plan

This project aim to reproduce Sora (Open AI T2V model), we wish the open source community contribute to this project.
MIT License
11.28k stars 1.01k forks source link

how to train the pretrained Latte model weights? #256

Open leonardodora opened 5 months ago

leonardodora commented 5 months ago

as the vae of opensora is different from Latte, the weights from latte could be able to use directly? Or your team train a latte model from scratch?

LinB203 commented 5 months ago

Yes, we use it directly. It adapts very quickly and the transformation is visible in about 500 steps. This is consistent with pixart-sigma.

leonardodora commented 5 months ago

Yes, we use it directly. It adapts very quickly and the transformation is visible in about 500 steps. This is consistent with pixart-sigma.

Thanks for your reply! But if we want to scale up the param number, what should we do first, you suggest? just train a larger new latte model?

LinB203 commented 5 months ago

Yes, we use it directly. It adapts very quickly and the transformation is visible in about 500 steps. This is consistent with pixart-sigma.

Thanks for your reply! But if we want to scale up the param number, what should we do first, you suggest? just train a larger new latte model?

I think a pixart-alpha should be retrained.