search

mit-han-lab / nunchaku

SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
https://hanlab.mit.edu/projects/svdquant
Apache License 2.0
351 stars 17 forks source link

is support controlnet and ip-adapter? #3

Open libai-lab opened 2 weeks ago

libai-lab commented 2 weeks ago

nice project,is support controlnet and ip-adapter?is there a demo

lmxyy commented 2 weeks ago

At https://svdquant.mit.edu, we have a demo for sketch-to-image, which is trained with img2img-turbo. As long as the add-on (e.g., image control, ip-adapter) is a LoRA, SVDQuant can support it without re-quantization. We will later release the tutorial on how to apply customized LoRA to our 4-bit base model.

peki12345 commented 2 weeks ago

At https://svdquant.mit.edu, we have a demo for sketch-to-image, which is trained with img2img-turbo. As long as the add-on (e.g., image control, ip-adapter) is a LoRA, SVDQuant can support it without re-quantization. We will later release the tutorial on how to apply customized LoRA to our 4-bit base model.

But neither ControlNet nor IP-Adapter is LORA, right?

libai-lab commented 2 weeks ago

At https://svdquant.mit.edu, we have a demo for sketch-to-image, which is trained with img2img-turbo. As long as the add-on (e.g., image control, ip-adapter) is a LoRA, SVDQuant can support it without re-quantization. We will later release the tutorial on how to apply customized LoRA to our 4-bit base model.

But neither ControlNet nor IP-Adapter is LORA, right?

no ,They are not Lorafor example， https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro,https://huggingface.co/XLabs-AI/flux-ip-adapter-v2

lmxyy commented 2 weeks ago

Currently, we only support the LoRA add-ons. I quickly checked the FLUX ControlNet and IP-Adapters. They introduce only a small number of additional parameters compared to the base model, just like LoRA. Even at 16-bit precision, their computational and parameter overhead remain minimal. We are considering adding support for these add-ons in our next release—stay tuned!