search

cubiq / Diffusers_IPAdapter

implementation of the IPAdapter models for HF Diffusers
MIT License
160 stars 6 forks source link

Integrating with `sd-webui-controlnet` A1111 extension to allow for multiple images per IP Adapter module #3

Closed josephrocca closed 10 months ago

josephrocca commented 10 months ago

I came across this video: https://www.youtube.com/watch?v=HtmIC6fqsMQ and this one: https://www.youtube.com/watch?v=7m9ZZFU3HWo and the results look great. IIUC, they're using multiple images with a single IP Adapter instance. So I went hunting for how to do this with A1111, but it looks like the most popular extension ( https://github.com/Mikubill/sd-webui-controlnet ) doesn't currently allow for multiple images.

I came across your comment here and am wondering if there's some way to integrate this with that A1111 extension?

I've not looked "under the hood" on IP adapter, this repo, or the controlnet extension, so I'm sorry if I've got some misunderstandings here! I'm very much toward the "end user" end of the spectrum.

cubiq commented 10 months ago

I had a very quick look at a1111 code, it looks very familiar, I don't see any issue with adding multiple images but I'm not familiar with their code. All they have to do is to patch the number of tokens in attn_processor with (number_of_images * ipadapter tokens)

josephrocca commented 10 months ago

Any chance you could submit a pull request? Even just a draft/sketch to show the rough idea, that someone with more experience with the repo could use as a guide? No worries if not - thank you for your work on this repo :pray:

cubiq commented 10 months ago

I'll see what I can do but I'm already working on the comfyui and diffusers versions, I don't have much time left for A1111