Purpose of Adapter Modules in training?

tencent-ailab / IP-Adapter

The image prompt adapter is designed to enable a pretrained text-to-image diffusion model to generate images with image prompt.

Apache License 2.0

4.5k stars 296 forks source link

Purpose of Adapter Modules in training? #274

Closed Bilal143260 closed 5 months ago

Bilal143260 commented 5 months ago

The class IPAdapter takes in image_proj_model and adapter_modules along with some other params in it's constructor. image_proj_model is used in forward method and returns ip_tokens which are then concatenated with text-embeddings and passed to unet. However, I don't seee adapter_modules being used in forward method yet it is passed to optimizer. Can you clarify what's the difference between two and why do we even need adapter_modules if we are not even passing it to unet?

xiaohu2015 commented 5 months ago

hi, used in attention processor