Abalation study with training with the unet frozen

[trideeprath]

Has there been any ablation study to understand the impact of freezing the unet and only training the clip encoder.

My understanding here is that if the unet is fixed this technique could be applied with other merged/lora based unets.

[foivospar]

Hi, the core module that needs tuning is the clip encoder, as it needs to be adapted to ID-embeddings. However, to achieve best results, we fine-tuned both the unet and clip encoder. I assume that some pre-trained LoRAs may still be compatible with our unet (as is the LCM-LoRA for example), but, in general, styled LoRAs that include both unet and clip weights are less likely to work, as our encoder is significantly altered from the original.

[trideeprath]

Is it possible to release the clip encoder with frozen unet understand and compare with approaches like ip-adapter that keep the unet frozen.