Can subject-driven or controllable generation pipelines be integrated with inpainting models?

Zeju1997 / oft

Official implementation of "Controlling Text-to-Image Diffusion by Orthogonal Finetuning".

https://oft.wyliu.com/

MIT License

280 stars 14 forks source link

Can subject-driven or controllable generation pipelines be integrated with inpainting models? #3

Closed i-amgeek closed 1 year ago

Zeju1997 commented 1 year ago

Hi, thank you for your interest in our project. I am not so familiar with the inpainting task, but to summarize our project, what we propose is a new fine-tuning method, that can be applied to all the tasks where LoRA can be applied. So if you have seen LoRA on the tasks you are working on, our method can also be used there.

I will add some examples in Readme of how to use our fine-tuning method on its own.

StephanPan commented 1 year ago

can i both use subject images and control signal such as detph as the input to finetune the model, so that the style and the stucture embedding can be both trained.