[Feature Request]: Implement X-Adapter

lllyasviel / stable-diffusion-webui-forge

GNU Affero General Public License v3.0

8.13k stars 796 forks source link

[Feature Request]: Implement X-Adapter #290

Open sashasubbbb opened 8 months ago

sashasubbbb commented 8 months ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

https://github.com/showlab/X-Adapter Code for X-Adapter is finally out.

X-Adapter enables plugins pretrained on old version (e.g. SD1.5) directly work with the upgraded Model (e.g., SDXL) without further retraining.

Is it possible to implement it into Forge?

Proposed workflow

Use 1.5 Loras on SDXL models.

Additional information

No response

huchenlei commented 8 months ago

@lllyasviel What is the best way to run 2 unets side by side now? It seems like the core logic for X-Adapter is mapping sd15 hidden states to SDXL hidden states (Add to original SDXL hidden states) in decode part of unet.

https://github.com/showlab/X-Adapter/blob/d5460d3baaa3e995c18dbf6680a843c8a3a9b3f9/pipeline/pipeline_sd_xl_adapter_controlnet_img2img.py#L1088-L1227

FurkanGozukara commented 8 months ago

Awesome following this topic

strawberrymelonpanda commented 8 months ago

@huchenlei Do they absolutely have to be side-by-side, or can they be loaded and unloaded one, then the other?

I ask because it doesn't want to run as-is on my 8GB of VRAM. It OOM'ed during the 2nd set of generation iterations. By the description / tutorial here by the author I was guessing it first generates using SD1.5 and then using SDXL, but I could be wrong.

I was able to make it work by changing every CUDA reference to CPU just for testing at a very slow 30~ minutes an image. Using inference.py --plugin_type "lora" with --adapter_guidance_start_list 0.7 and an "old pencil sketch" style LoRA, I get a decent effect.

Gushousekai195 commented 7 months ago

I need this... right now

shitianfang commented 7 months ago

I need this... right now

huchenlei commented 7 months ago

According to my testing X-Adapter result is similar to running HR fix with SD15 model doing low-res pass and SDXL model doing highres pass.

See https://github.com/Mikubill/sd-webui-controlnet/issues/2652#issuecomment-1972292154.

metapea commented 5 months ago

There also SD-Latent-Interposer, as an alternative And the dev says a A1111 version can be implemented.