search

showlab / X-Adapter

[CVPR 2024] X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model
https://showlab.github.io/X-Adapter/
Apache License 2.0
740 stars 43 forks source link

Is SD15 base pass to T0 manatory? #23

Open huchenlei opened 8 months ago

huchenlei commented 8 months ago

I am trying to implement X-Adapter in A1111 here https://github.com/huchenlei/sd-webui-xadapter/. As A1111 does not support running 2 denosing unet side by side (Or change this need more engineering effort I would like to commit), I decided to make the SD15 pass run before the SDXL pass. Hidden state outputs of last 3 decoder blocks are stored, and then applied back onto the SDXL model via xadapter. Implementation can be found here: https://github.com/huchenlei/sd-webui-xadapter/blob/main/scripts/xadapter.py

I am omitting the SD15 base pass from T0 for simplicity. However, the result is not satisfactory. Am I missing something? Is the SD15 base pass manatory?

Semolce9 commented 8 months ago

Hi. Thank you for trying it in A1111. SD1.5's last three decoder's output changes after T0 and these changing values will guide sdxl in the following steps. Store it and meke it unchanged will affect the result.

huchenlei commented 8 months ago

In the code I am storing the sd15 hidden states for each timesteps.