search

thu-ml / cond-image-leakage

Official implementation for "Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model" (NeurIPS 2024)
Apache License 2.0
219 stars 25 forks source link

About the degaration results of training Dynamicrafter #8

Open leoisufa opened 2 months ago

leoisufa commented 2 months ago

Thanks for you wonderful work!!!

I try to reproduce the training results based on your Dynamicrafter code. However, I got some worse training results., like:

https://github.com/user-attachments/assets/ab2d4653-239f-4acf-a002-55df432fafc1

https://github.com/user-attachments/assets/d56a3f32-e7ed-46e8-9ed6-07aff0e8223c

https://github.com/user-attachments/assets/f1081599-4e8d-4613-8210-4470908b725d

I only chaged the batch size from 2 to 8 in the train_512.yaml, because I use the 8 A800-80G GPU, and I follow the setting in your paper, the total batch_size is 88=64. Above video results are generate from the checkpoint in 10000 steps. I find that you just train Dynamicrafter for 20,000 steps in your paper, I guess the checkpoint of 10000 steps should not generate so unsatisfatory results. Can you give some advice for reproducing your results?

zhuhz22 commented 1 month ago

Hi @leoisufa , Thank you for your kind words and sorry for replying late. The reason for the degaration lies in examples/DynamiCrafter/lvdm/models/ddpm3d.py#45, where mu_max=4 should be mu_max=1, which is corresponding to the paper. I'm really sorry that I changed this parameter during the attempt of adjusting parameters in the 1024 resolution training, and forgot to correct it back. Now it's fixed in th repo.