Open ArielReplicate opened 1 year ago
Hi @ArielReplicate, the ddim_steps
parameter indicates the number of backward sampling steps used during the inversion (i.e. from latent noise to image), while the fixed variable ddim_inversion_steps = 999
indicates the number of DDIM forward steps used in the inversion (i.e. from image to latent noise), so they can have differing values. We found the most reliable configuration for both parameters to be the full sampling steps (=999), where we get the best reconstruction of the structure image.
Note that for real images, the timesteps at which features are saved are determined by the save_feature_timesteps
parameter, which determines the num_ddim_sampling_steps
that can be used during the translation (num_ddim_sampling_steps = save_feature_timesteps
).
Thanks for the elaboration!
It seem though that ddim_steps is not related to inversion only to sampling after the inversion https://github.com/MichalGeyer/plug-and-play/blob/ce9e4e23c24b4241cb9ecae9ea0b090c70871870/run_features_extraction.py#L229
Is it using the fact that model.num_timesteps is set to 1000?
When I try to change the ddim_inversion_steps to 50 I got this error:
z_enc, _ = sampler.encode_ddim(init_latent, num_steps=exp_config.ddim_inversion_steps,
File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/src/ldm/models/diffusion/ddim.py", line 281, in encode_ddim
img, _ = self.reverse_ddim(img, t, t_next=steps[i+1] ,c=conditioning, unconditional_conditioning=unconditional_conditioning, unconditional_guidance_scale=unconditional_guidance_scale)
File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/src/ldm/models/diffusion/ddim.py", line 306, in reverse_ddim
a_next = torch.full((b, 1, 1, 1), alphas[t_next], device=device) #a_next = torch.full((b, 1, 1, 1), alphas[t + 1], device=device)
IndexError: index 1007 is out of bounds for dimension 0 with size 1000
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x86da6c]
I'm trying to speed things up (each of the inversion/sampling take ~1.5 minutes on a A100 instance) to speed up the
you are correct, ddim_steps
is only used during the backward sampling, which actually isn't a part of the inversion process (sorry for the misinterpretation in the previous answer). Notice that in ddim.sample
, the model scheduler timesteps are being reset to ddim_steps
.
The error you pointed out seems to be due to an incorrect calculation of inversion sampling steps in ddim.encode_ddim
in case ddim_inversion_steps = 50
.
We will work on fixing this issue, thanks for reporting it !
Hi it seems the number of DDim steps for inversion is fixed : https://github.com/MichalGeyer/plug-and-play/blob/ce9e4e23c24b4241cb9ecae9ea0b090c70871870/run_features_extraction.py#L224
Is there a specific reason why you only exposed the number of parameters for sampling the features (ddim_steps: 999 in feature extraction .yaml config files)
How related are the two parameters (i.e ddim_inversion_steps and exp_config.config.ddim_steps ) does a change in one requires changing the other? I think the inversion could be good enough with 50 ddim_steps ?