search

zyxElsa / ProSpect

Official implementation of the paper "ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models"(SIGGRAPH Asia 2023)
Apache License 2.0
126 stars 10 forks source link

attribute aware img2img workflow #9

Open freLorbeer opened 11 months ago

freLorbeer commented 11 months ago

Hi thanks for your work!

I am a bit confused about the explicit workflow for attribute manipulation in the img2img generation process.

how can I insert a material/layout/content from an image in the generation process since there is no style_dir param? Do I need to finetune/train the model on only one image an then use the prompts according to the T2I prompts in the Read.Me (* gets replaced by the learned embedding)? What is the general idea behind the finetuning of the model?

zyxElsa commented 11 months ago

Hi. The target concept is injected by replacing the * with the learned token embedding provided by the hypernet. Only the hypernet is trainable during training, and the LDM model is always frozen. The framework works in a per-image-per-hypernet manner.

surfingnirvana commented 3 months ago

Thank you. How can i train the hypernet with a specific token embedding? Where is the config file to define which attributes to transfer over?