search

pesser / stable-diffusion

MIT License
1.03k stars 401 forks source link

About fine-tuning for inpainting #24

Open cats-food opened 2 years ago

cats-food commented 2 years ago

Huge thanks for your code contribution first!

I used your config file "v1-finetune-for-inpainting-laion-iaesthe.yaml" to fine-tune the model for text-conditioned inpainting. The dataset I used is this subset of the Liaon dataset.

It turns out the results finally become the naive inpainting (simply fills the missing region), and were no longer controlled by the text conditioning as the training proceeds (as shown below, the txt prompt is "a cat on the bench", but no cat appears). Maybe i miss some tricks, I wonder did you meet the same issue when you trained the model?

image

Thank you in advance :)

franklinosei commented 2 years ago

Hello @Yang-Shiyuan,

Have you found a fix for that? I've also been trying to finetune the inpainting model for a specific problem I have but I can't seem to get the setup right. Do you mind sharing your training script with me? Just to help me get started. Thanks

zaryabmakram commented 1 year ago

Hey @Yang-Shiyuan,

Can you describe the exact steps you took to achieve these results in a bit more detail? It would be really helpful to know the specifics of your training setup.