Adding more attention layers for extra conditioning

tencent-ailab / IP-Adapter

The image prompt adapter is designed to enable a pretrained text-to-image diffusion model to generate images with image prompt.

Apache License 2.0

5.08k stars 331 forks source link

Adding more attention layers for extra conditioning #278

Open Bilal143260 opened 8 months ago

Bilal143260 commented 8 months ago

Is it a good idea to add more attention layers for extra conditioning? Or adding controlnet with IP_adapter would the job better?

xiaohu2015 commented 8 months ago

yes, see instantid (using ip-adapter + controlnet)

Bilal143260 commented 8 months ago

Why is using ip-adapter+controlnet over adding more attention layers a better choice?

xiaohu2015 commented 8 months ago

Why is using ip-adapter+controlnet over adding more attention layers a better choice?

I am not sure which is better, maybe should do some experiments

Bilal143260 commented 8 months ago

If I add more attention layers, what changes would I need to make in inference code?

zhanglaplace commented 8 months ago

yes, see instantid (using ip-adapter + controlnet)

ip-adapter-face + controlnet-openpose 效果看着不太好，是需要重新finetune吗

` controlnet_model_path = "stablediffusion/lllyasviel/control_v11p_sd15_openpose"

controlnet = ControlNetModel.from_pretrained(controlnet_model_path, torch_dtype=torch.float16)

pipe = StableDiffusionControlNetPipeline.from_pretrained( base_model_path, controlnet=controlnet, torch_dtype=torch.float16, scheduler=noise_scheduler, vae=vae, feature_extractor=None, safety_checker=None ) `