search

pixeli99 / SVD_Xtend

Stable Video Diffusion Training Code and Extensions.
482 stars 45 forks source link

Finetune SVD with discrete time noise scheduler #35

Open HL-hanlin opened 4 months ago

HL-hanlin commented 4 months ago

Hi, thanks for your great training code for SVD!

When I switch the default SVD sampler into discrete noise scheduler, the generated video looks very bad. May I ask which layers/parameters of SVD should I finetune to make it compatible with discrete time noise scheduler? Or are there some ways that can transform the SVD noisy_latents (which is equal to latents + sigma * noise, with large sigma values) into corresponding noisy_latents (with smaller sigmas) used to train on discrete noise schedulers?

Thanks!

pixeli99 commented 4 months ago

You should need the FT timestep condition, you can refer to this part of the SVD paper for some help image

HL-hanlin commented 4 months ago

Got it! Thanks for the pointer!