search

ShivamShrirao / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch
https://huggingface.co/docs/diffusers
Apache License 2.0
1.89k stars 505 forks source link

Prompt per image (like regular fine-tuning) #185

Closed gleb-akhmerov closed 1 year ago

gleb-akhmerov commented 1 year ago

It seems that in order to turn Dreambooth training into regular fine-tuning, all you need to do is use a separate prompt for each image. At least, as far as I understand. :smile:

I've been experimenting with this option, and the resulting model seems to be more controllable - during inference, generated images more closely followed the prompt and also I was able to edit the prompt more freely. It felt like the model was no longer "glued" to the instance prompt and had more understanding of the prompt's parts.

Inspired by https://github.com/andreasjansson/monkey-island-sd, specifically this change.

G-force78 commented 1 year ago

yeah I was wondering how the script knows to associate different instance prompts with the different instance images, it has a field for multiple concepts but it seems they would just get mixed up if they aren't associated with the images.

ShivamShrirao commented 1 year ago

Sorry for the delay, missed it.