search

huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
25.16k stars 5.2k forks source link

IPAdapter attention layers run with requires_grad=True for parameters #9336

Open darshats opened 2 weeks ago

darshats commented 2 weeks ago

Describe the bug

When IP adapter is loaded, the IPAdapterAttnProcessor2_0 class is not set with training=False. Didnt see a huge difference in memory if its manually set to false on those processors, but it looks like a miss? Same for image_projection_layers

Reproduction

pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="sdxl_models", weight_name=["ip-adapter-plus_sdxl_vit-h.safetensors"]*1, use_safetensors=True)
list(pipeline.unet.encoder_hid_proj.image_projection_layers.parameters())
for k, v in pipeline.unet.attn_processors.items():
    if isinstance(v, IPAdapterAttnProcessor2_0) and v.training:
        print(len(list(v.parameters())))

Logs

No response

System Info

diffusers 0.28.2 with sdxl

Who can help?

No response

tolgacangoz commented 2 weeks ago

See this decorator on top of the __call__ method at all the pipelines: torch.no_grad(). However, if one wants to use IPAdapterAttnProcessor2_0 or a similar component as a standalone component, one needs to consider this.