search

open-mmlab / PowerPaint

[ECCV 2024] PowerPaint, a versatile image inpainting model that supports text-guided object inpainting, object removal, image outpainting and shape-guided object inpainting with only a single model. 一个高质量多功能的图像修补模型，可以同时支持插入物体、移除物体、图像扩展、形状可控的物体生成，只需要一个模型
https://powerpaint.github.io/
MIT License
364 stars 18 forks source link

About optimize Pctxt #10

Open shouwangzhe200 opened 6 months ago

shouwangzhe200 commented 6 months ago

image According to above equation, is it only the learnable prompt Pctxt trained during finetune? Or the paramters of Unet are also finetuned at the same time?

icsl-Jeon commented 3 months ago

I am also very curious..

lijiaxing0213 commented 2 months ago

Optimization should only be performed on the embeddings of the task prompt, while the parameters of the unet are frozen. You can refer to the implementation of the EmbeddingLayerWithFixes class at https://mmagic.readthedocs.io/en/latest/autoapi/mmagic/models/editors/disco_diffusion/clip_wrapper/index.html#mmagic.models.editors.disco_diffusion.clip_wrapper.EmbeddingLayerWithFixes.