Plans to support LoRas?

[levi]

Since IPAdapter replaces the attention processor, loading a LoRa via diffusers has not been possible in my testing.

Do you have plans to add support for LoRas in the current IPAdapter design? Is this an area of investigation for future versions?

[budui]

You can use LoRA and IP-adapter at the same time with diffusers 0.21.0.dev

diffusers 0.21.0(current in master branch) replace nn.Linear with LoRACompatibleLinear, so LoRA will not need attention processor.

#https://github.com/huggingface/diffusers/blob/19edca82f1ff194c07317369a92b470dbae97f34/src/diffusers/models/attention_processor.py#L1471-L1479
       deprecate(
            self_cls_name,
            "0.26.0",
            (
                f"Make sure use {self_cls_name[4:]} instead by setting"
                "LoRA layers to `self.{to_q,to_k,to_v,to_out[0]}.lora_layer` respectively. This will be done automatically when using"
                " `LoraLoaderMixin.load_lora_weights`"
            ),
        )

[blx0102]

@budui Hi, I'm using diffusers==0.21.4 but still get error when using a pipeline with lora loaded. The traceback is :

Traceback (most recent call last):
  File "/mnt/disk1/code/ip-adapter/infer-face-lora.py", line 45, in <module>
    ip_model_plus = IPAdapterPlus(pipe, image_encoder_path, ip_ckpt_plus_face, device, num_tokens=16)
  File "/mnt/disk1/code/ip-adapter/ip_adapter/ip_adapter.py", line 53, in __init__
    self.load_ip_adapter()
  File "/mnt/disk1/code/ip-adapter/ip_adapter/ip_adapter.py", line 92, in load_ip_adapter
    ip_layers = torch.nn.ModuleList(self.pipe.unet.attn_processors.values())
  File "/home/.local/lib/python3.10/site-packages/diffusers/models/modeling_utils.py", line 210, in __getattr__
    return super().__getattr__(name)
  File "/mnt/disk1/anaconda3/envs/py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1614, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'UNet2DConditionModel' object has no attribute 'attn_processors'. Did you mean: 'set_attn_processor'?

Do you have any idea how to fix it?

[oloflindh]

Hey @budui! I'm also having trouble applying a LoRA with an IP pipeline (using diffusers==0.21.2). Are there any docs or examples of how to correctly combine the SDXL IPAdapter with LoRAs? Thanks!

[chenxinhua]

pipe = StableDiffusionPipeline.from_pretrained( base_model_path, torch_dtype=torch.float16, scheduler=noise_scheduler, vae=vae, feature_extractor=None, safety_checker=None )

lora_model_path="/mnt/dolphinfs/ssd_pool/docker/user/hadoop-fincv/chenxinhua03/projects/AIGC/experiments/style_lora" weight_name = "Watercolor_Painting_by_vizsumit.safetensors" pipe.load_lora_weights(lora_model_path, weight_name=weight_name, adapter_name="style") pipe.set_adapters(["style"], adapter_weights=[0.4])

ip_model = IPAdapterFull(pipe, image_encoder_path, ip_ckpt, device, num_tokens=257) pil_image = Image.open("images/3.jpg") prompt = "A watercolor paint"

install peft, and this code works for me

[Daming-TF]

pipe = StableDiffusionPipeline.from_pretrained( base_model_path, torch_dtype=torch.float16, scheduler=noise_scheduler, vae=vae, feature_extractor=None, safety_checker=None )

lora_model_path="/mnt/dolphinfs/ssd_pool/docker/user/hadoop-fincv/chenxinhua03/projects/AIGC/experiments/style_lora" weight_name = "Watercolor_Painting_by_vizsumit.safetensors" pipe.load_lora_weights(lora_model_path, weight_name=weight_name, adapter_name="style") pipe.set_adapters(["style"], adapter_weights=[0.4])

ip_model = IPAdapterFull(pipe, image_encoder_path, ip_ckpt, device, num_tokens=257) pil_image = Image.open("images/3.jpg") prompt = "A watercolor paint"

install peft, and this code works for me

Hello, what is your diffusers version

[Jun-Pu]

pipe = StableDiffusionPipeline.from_pretrained( base_model_path, torch_dtype=torch.float16, scheduler=noise_scheduler, vae=vae, feature_extractor=None, safety_checker=None )

lora_model_path="/mnt/dolphinfs/ssd_pool/docker/user/hadoop-fincv/chenxinhua03/projects/AIGC/experiments/style_lora" weight_name = "Watercolor_Painting_by_vizsumit.safetensors" pipe.load_lora_weights(lora_model_path, weight_name=weight_name, adapter_name="style") pipe.set_adapters(["style"], adapter_weights=[0.4])

ip_model = IPAdapterFull(pipe, image_encoder_path, ip_ckpt, device, num_tokens=257) pil_image = Image.open("images/3.jpg") prompt = "A watercolor paint"

install peft, and this code works for me

Yes! Installing peft solves the prob!

[Jun-Pu]

pipe = StableDiffusionPipeline.from_pretrained( base_model_path, torch_dtype=torch.float16, scheduler=noise_scheduler, vae=vae, feature_extractor=None, safety_checker=None ) lora_model_path="/mnt/dolphinfs/ssd_pool/docker/user/hadoop-fincv/chenxinhua03/projects/AIGC/experiments/style_lora" weight_name = "Watercolor_Painting_by_vizsumit.safetensors" pipe.load_lora_weights(lora_model_path, weight_name=weight_name, adapter_name="style") pipe.set_adapters(["style"], adapter_weights=[0.4]) ip_model = IPAdapterFull(pipe, image_encoder_path, ip_ckpt, device, num_tokens=257) pil_image = Image.open("images/3.jpg") prompt = "A watercolor paint" install peft, and this code works for me

Hello, what is your diffusers version

As far as I know diffusers==0.26.3 does work