Open alexblattner opened 1 year ago
Hi, there
(1) I might be able to revise it directly if it's open-source. If so, please show me the repo link.
(2) To revise the multi-control (mc) pipeline, you can refer to the pipeline implementation
class PaintWithWord_StableDiffusionPipeline(StableDiffusionPipeline):
in 'paint_with_words.py' at https://github.com/lwchen6309/paint-with-words-sd/blob/ae75a8f6d1279c501c17a2482164571962761816/paint_with_words/paint_with_words.py#L513 especially for the denoising step of call function at https://github.com/lwchen6309/paint-with-words-sd/blob/ae75a8f6d1279c501c17a2482164571962761816/paint_with_words/paint_with_words.py#L783
# 7. Denoising loop
num_warmup_steps = len(timesteps) - num_inference_steps * self.scheduler.order
with self.progress_bar(total=num_inference_steps) as progress_bar:
for i, t in enumerate(timesteps):
step_index = (self.scheduler.timesteps == t).nonzero().item()
sigma = self.scheduler.sigmas[step_index]
latent_model_input = self.scheduler.scale_model_input(latents, t)
# _t = t if not is_mps else t.float()
encoder_hidden_states.update({
"SIGMA": sigma,
"WEIGHT_FUNCTION": weight_function,
})
noise_pred_text = self.unet(
latent_model_input,
t,
encoder_hidden_states=encoder_hidden_states,
).sample
(a) As you can see here, the unet receive a conditional tensor
encoder_hidden_states
, which used to be a tensor but now replaced by a dict that consists of sigma, weight_funcion and the original tensor.(b) You also have to replace the forward function of cross attention as at https://github.com/lwchen6309/paint-with-words-sd/blob/ae75a8f6d1279c501c17a2482164571962761816/paint_with_words/paint_with_words.py#L539 You can add (1) and (2) to mc pipeline to combine pww and mc pipeline.
(3) I'd also recommend you to try (latent couple)[https://github.com/opparco/stable-diffusion-webui-two-shot], which simply modify the unet input by adding text weighted map. Using this, you don't even need to inject the forward function of cross attention module but directly revise the denoising steps.
I'll be completely honest with you. I don't know how either multicontrolnet or paintwithwords work beyond some theory. For multicontrolnet all I know is that it uses multiple controlnet models to generate something (don't know how they're mixed together or how the weights are taken into account) and for paintwithwords I have no idea how you tell it to focus prompts on specific parts of an image (i know there's color mapping but not how it works).
Are we just adding more and more weights and it works? But paintwithwords doesn't use weights and won't things get corrupted eventually with all those models and weights?
In that case, I'm afraid it would be hard for you to revise the code without knowing the theory behind them, which is probably why (a) and (b) are difficult to understand. To mix the mc pipeline and pww, it would be essential to know how they work. I'm afraid I cannot help with this unfortunately.
Hi, I'm not at your level and was wondering how I could add paint with words to my multicontrolnet pipeline. Here's code that works for example (partial):
I would appreciate your input on it and am ready to pay if necessary