search

lllyasviel / Omost

Your image is almost there!
Apache License 2.0
6.69k stars 401 forks source link

IP-Adapter support? #46

Open jadechip opened 3 weeks ago

jadechip commented 3 weeks ago

Would be cool to have support for IP-Adapter added. As the pipeline inherits from StableDiffusionXLImg2ImgPipeline which supports ip-adapter, I assume it should be quite simple to add support for ip-adapter? ...but give the custom implementation of the sampling process, I am not so sure, perhaps something like this?

if ip_adapter_image is not None or ip_adapter_image_embeds is not None:
    image_embeds = self.prepare_ip_adapter_image_embeds(
        ip_adapter_image,
        ip_adapter_image_embeds,
        device,
        batch_size * num_images_per_prompt,
        self.do_classifier_free_guidance,
    )

if ip_adapter_image is not None or ip_adapter_image_embeds is not None:
    added_cond_kwargs["image_embeds"] = image_embeds
jadechip commented 3 weeks ago

I took a quick stab at it, but I am not sure if this is right? #56 Need to run some tests...

jadechip commented 3 weeks ago

Currently unable to load the ip-adapter image, seems like the weights of the clip encoder isn't loaded on the gpu? 

 File "/home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py", line 774, in encode_image
    image_embeds = self.image_encoder(image).image_embeds

`RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same`