search

cubiq / Diffusers_IPAdapter

implementation of the IPAdapter models for HF Diffusers
MIT License
147 stars 6 forks source link

Diffusers with Multiple IPadapters? #2

Open hajiarab1988 opened 9 months ago

hajiarab1988 commented 9 months ago

Hi and thank you for your amazing job. I'm aware that with comfyui, it's possible to use multiple IPAdapter instances. However, I'm looking for insights on how to achieve this with diffusers and StableDiffusionPipelines.

  1. Can I use two IPAdapter instances within a single StableDiffusionPipelines pipeline in the diffusers library?
  2. If possible, how can I achieve this? thank you
hajiarab1988 commented 9 months ago

@cubiq

cubiq commented 9 months ago

there should be no need to do that, you can send multiple images and set different weights to them.

why do you want to use two instances?

hajiarab1988 commented 9 months ago

there should be no need to do that, you can send multiple images and set different weights to them.

why do you want to use two instances?

@cubiq Thanks for your response. I want to use two IPAdapter instances in a single pipeline to combine the results of face ipadapter and plus ipadapter.This setup helps me achieve consistent character faces with consistent outfit adoption. Is it possible to do this, or do you recommend an alternative approach? Your insights are highly appreciated.

cubiq commented 8 months ago

I need to do a small change to the code in order to support that. I'll work on it when I have a little time

xiaohu2015 commented 8 months ago

https://github.com/huggingface/diffusers/pull/5713

cubiq commented 8 months ago

that's great! I hope it gets integrated into diffusers directly so I can close this repo :smile:

at this point though my comfyui implementation has so many more features that I don't see why you should use anything else :smile: I'll check their code, but I'm sure they wouldn't accept many of my modifications

xiaohu2015 commented 8 months ago

that's great! I hope it gets integrated into diffusers directly so I can close this repo 😄

at this point though my comfyui implementation has so many more features that I don't see why you should use anything else 😄 I'll check their code, but I'm sure they wouldn't accept many of my modifications

You repo is great! I think you can give some suggestions!

cubiq commented 8 months ago

You repo is great! I think you can give some suggestions!

I checked their code, they do things the "diffusers way" but they don't add any of the new features that I've added... I'll see if they accept suggestions

cubiq commented 8 months ago

well I tried... I don't think it is going to happen... soon :smile:

usamaehsan commented 7 months ago

"I need to do a small change to the code in order to support that. I'll work on it when I have a little time" @cubiq any update on multiple ip-adapters?

cubiq commented 7 months ago

yeah I'll work on that and more features in the coming days.

diffusers now supports IPA natively but they skipped on all the cool stuff... maybe I can hook to their implementation and just add what is missing

ynie commented 7 months ago

+1 on this. I would really want to use the regular ip_adapter for the style rendering for maybe [1-0.8] and then face ip_adapter[0.8-1] for face consistency.

cubiq commented 7 months ago

sorry guys tencent labs just released FaceID and got side tracked! I'm gonna work on this too, no worries

ynie commented 7 months ago

Thanks! Let me know if there’s anything I can help. Happy to test as well.

whiterose199187 commented 6 months ago

hello @cubiq

Is there any progress on this? Happy to help test if you need..

thibaudart commented 6 months ago

hey @cubiq i try to combine two ipadapters. one for face, one for style

hey @cubiq prompt_embeds, negative_prompt_embeds, pooled_prompt_embeds, negative_pooled_prompt_embeds = ip_adapter.get_prompt_embeds(...)

prompt_embeds2, negative_prompt_embeds2, pooled_prompt_embeds2, negative_pooled_prompt_embeds2 = ip_adapter2.get_prompt_embeds(...)

then combine the two

prompt_embeds = prompt_embeds + prompt_embeds but it doesn't work:

RuntimeError: The size of tensor a (253) must match the size of tensor b (93) at non-singleton dimension 1

have you any pointer to help?

cubiq commented 6 months ago

I haven't forgotten about this, guys! I already started working on a complete rewrite to add all the missing options.

thibaudart commented 6 months ago

thanks for the update! btw when you work on it, if you could made a sample to show how to combine the multi ipadapters with multi controlnet in the same script. it ll be so helpful.

jyoung105 commented 4 months ago

@thibaudart hi, I want to ask you how it goes. If you adopt 2 ip adapters, will you divide the prompt to get an average or just leave it as added one?