search

lllyasviel / ControlNet

Let us control diffusion models!
Apache License 2.0
30.4k stars 2.73k forks source link

how to use DDIM inversion during inference? #359

Open XiaoyuShi97 opened 1 year ago

XiaoyuShi97 commented 1 year ago

Hi, thanks for releasing this amazing project. From my best understanding, current demo code does not use DDIM inversion during inference as the x_T variable is None in the following line? If so, could you please provide some suggestions on how to implement DDIM inversion to better use some input images as prior? Thanks!

https://github.com/lllyasviel/ControlNet/blob/3d25e488c7e6c2efdfe94f1673f2de261c662902/cldm/ddim_hacked.py#L132-L133

LiuShiyu95 commented 1 year ago

Hi @btwbtm , may I ask if you have recently conducted some research using DDIM conversion and controllnet? I tried it but it was not successful. There was a problem with the code during gradient feedback，so I would like to consult with you.

XiaoyuShi97 commented 1 year ago

Hi @LiuShiyu95 , I made some attempts. To my best understanding, the following function is DDIM inversion if you pass in the ddim schedule? https://github.com/lllyasviel/ControlNet/blob/main/cldm/ddim_hacked.py#L234

LiuShiyu95 commented 1 year ago

I use DDIM inversion to find the noise and can't control the image texture to stay the same, and I get some errors when I try to use Null-text optimization. How do you keep the texture consistent during your attempts? @btwbtm

XiaoyuShi97 commented 1 year ago

I use DDIM inversion to find the noise and can't control the image texture to stay the same, and I get some errors when I try to use Null-text optimization. How do you keep the texture consistent during your attempts? @btwbtm

DDIM inversion in theory can not reconstruct exactly the same image. I did not try other inversion methods. What kind of errors do you encounter?

LiuShiyu95 commented 1 year ago

I use DDIM inversion to find the noise and can't control the image texture to stay the same, and I get some errors when I try to use Null-text optimization. How do you keep the texture consistent during your attempts? @btwbtm

DDIM inversion in theory can not reconstruct exactly the same image. I did not try other inversion methods. What kind of errors do you encounter?

Thank you for your reply. I just changed the pipeline in null text conversion from SD to Controlnet, which will cause autograd errors. I just started learning about deep learning and SD models, and my code ability is poor. It should be my own problem. I will try other method, thanks again.

pikeyang commented 1 year ago

Do you have any new findings？