Diffusers with Multiple IPadapters?

[hajiarab1988]

Hi and thank you for your amazing job. I'm aware that with comfyui, it's possible to use multiple IPAdapter instances. However, I'm looking for insights on how to achieve this with diffusers and StableDiffusionPipelines.

1. Can I use two IPAdapter instances within a single StableDiffusionPipelines pipeline in the diffusers library?
2. If possible, how can I achieve this? thank you

[hajiarab1988]

@cubiq

[cubiq]

there should be no need to do that, you can send multiple images and set different weights to them.

why do you want to use two instances?

[hajiarab1988]

there should be no need to do that, you can send multiple images and set different weights to them.

why do you want to use two instances?

@cubiq Thanks for your response. I want to use two IPAdapter instances in a single pipeline to combine the results of face ipadapter and plus ipadapter.This setup helps me achieve consistent character faces with consistent outfit adoption. Is it possible to do this, or do you recommend an alternative approach? Your insights are highly appreciated.

[cubiq]

I need to do a small change to the code in order to support that. I'll work on it when I have a little time

[xiaohu2015]

https://github.com/huggingface/diffusers/pull/5713

[cubiq]

that's great! I hope it gets integrated into diffusers directly so I can close this repo :smile:

at this point though my comfyui implementation has so many more features that I don't see why you should use anything else :smile: I'll check their code, but I'm sure they wouldn't accept many of my modifications

[xiaohu2015]

that's great! I hope it gets integrated into diffusers directly so I can close this repo 😄

at this point though my comfyui implementation has so many more features that I don't see why you should use anything else 😄 I'll check their code, but I'm sure they wouldn't accept many of my modifications

You repo is great! I think you can give some suggestions!

[cubiq]

You repo is great! I think you can give some suggestions!

I checked their code, they do things the "diffusers way" but they don't add any of the new features that I've added... I'll see if they accept suggestions

[cubiq]

well I tried... I don't think it is going to happen... soon :smile:

[usamaehsan]

"I need to do a small change to the code in order to support that. I'll work on it when I have a little time" @cubiq any update on multiple ip-adapters?

[cubiq]

yeah I'll work on that and more features in the coming days.

diffusers now supports IPA natively but they skipped on all the cool stuff... maybe I can hook to their implementation and just add what is missing

[ynie]

+1 on this. I would really want to use the regular ip_adapter for the style rendering for maybe [1-0.8] and then face ip_adapter[0.8-1] for face consistency.

[cubiq]

sorry guys tencent labs just released FaceID and got side tracked! I'm gonna work on this too, no worries

[ynie]

Thanks! Let me know if there’s anything I can help. Happy to test as well.

[whiterose199187]

hello @cubiq

Is there any progress on this? Happy to help test if you need..

[thibaudart]

hey @cubiq i try to combine two ipadapters. one for face, one for style

hey @cubiq prompt_embeds, negative_prompt_embeds, pooled_prompt_embeds, negative_pooled_prompt_embeds = ip_adapter.get_prompt_embeds(...)

prompt_embeds2, negative_prompt_embeds2, pooled_prompt_embeds2, negative_pooled_prompt_embeds2 = ip_adapter2.get_prompt_embeds(...)

then combine the two

prompt_embeds = prompt_embeds + prompt_embeds but it doesn't work:

RuntimeError: The size of tensor a (253) must match the size of tensor b (93) at non-singleton dimension 1

have you any pointer to help?

[cubiq]

I haven't forgotten about this, guys! I already started working on a complete rewrite to add all the missing options.

[thibaudart]

thanks for the update! btw when you work on it, if you could made a sample to show how to combine the multi ipadapters with multi controlnet in the same script. it ll be so helpful.

[jyoung105]

@thibaudart hi, I want to ask you how it goes. If you adopt 2 ip adapters, will you divide the prompt to get an average or just leave it as added one?