Why are you concatenating zeros to the control residual?

google / style-aligned

Official code for "Style Aligned Image Generation via Shared Attention"

Apache License 2.0

1.19k stars 86 forks source link

Why are you concatenating zeros to the control residual? #24

Closed CesarERamosMedina closed 7 months ago

CesarERamosMedina commented 7 months ago

Hi! Great paper and great code. While looking at the pipeline calls I noticed that you are concatenating zeroes tensors to the control residuals when running SDXL ControlNet, and I am not sure why after reading the paper+code. Is this needed only for SDXL? I am adapting the controlnet style transfer for SD1 and I don't seem to need to make that modification to the pipeline call function.

CesarERamosMedina commented 7 months ago

It is to apply the output of the controlnet to both the conditional and unconditional embeddings, and keep the unconditional ones unchanged