search

huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
25.34k stars 5.25k forks source link

[FR] Multiclass Finetuning with Dreambooth #3309

Closed mauricio-repetto closed 1 year ago

mauricio-repetto commented 1 year ago

Is your feature request related to a problem? Please describe.

Hi! first of all thanks for such a great tool and all your hard work. Well at this moment I'm working with a partner @fededemo on a thesis related to data augmentation using synthetic images with generative models. We were fintuning things with dreambooth using some scripts from this repo which allows us to finetune two classes simultaneously by using a parameter call concept_list I would like if possilbe to add this kind of capabilty to the official scripts.

The concept list work this way concepts_list = [ { "instance_prompt": "photo of zwx dog", "class_prompt": "photo of a dog", "instance_data_dir": "/content/data/zwx", "class_data_dir": "/content/data/dog" }, { "instance_prompt": "photo of ukj person", "class_prompt": "photo of a person", "instance_data_dir": "/content/data/ukj", "class_data_dir": "/content/data/person" } ]

NOTE: In particular something that this script didn't take into account was to also consider sample_batch_size and save_sample_prompt within this dictionary for the additional classes since eventually someone would want to save different prompts and number of images for each class.

Describe the solution you'd like

We would like to see the above implemented, why? well we would like to see if the DM is capable of give better attention to the differences between objects by finetuning at the same time two different concepts that belong to the same class, for instance an image of a pneumonia xray and an image of a penoumoconiosis xray

Describe alternatives you've considered

Use the alternative scripts that we have being using so far, with the drawback that there aren't official scripts :'(

Additional context

-

patrickvonplaten commented 1 year ago

Could we maybe move this into the research_projects folder: https://github.com/huggingface/diffusers/tree/main/examples/research_projects and be maintained by you? :-)

mauricio-repetto commented 1 year ago

Hi @patrickvonplaten I can only promess that I'll try :-) Thanks in advance for always replying back

mauricio-repetto commented 1 year ago

Hello @patrickvonplaten, we spent some time on this and finally we have implemented it in the following pr: https://github.com/huggingface/diffusers/pull/3807