Fine tuning SDXL inpainting

[EnricoBeltramo]

Is your feature request related to a problem? Please describe. Training with LORA of SDXL refiner for inpainting

Describe the solution you'd like Is possibile to modify the solution proposed in: https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_sdxl.md in order to train refiners model too (i.e. for fine tune inpainting models)?

[sayakpaul]

How common is fine-tuning the refiner? I am yet to see some promising results from that. Plus it introduces a two-stage training pipeline I think:

• First LoRA on the base.
• Then do LoRA on the refiner.

Might be a complicated workflow.

[patrickvonplaten]

@williamberman is working on it I believe

[williamberman]

Yep! inpainting for sdxl is a wip here https://github.com/huggingface/diffusers/pull/4746

[abdellah-lamrani-alaoui]

Thanks a lot for the PR, will be super useful. Quick question, do you plan on training and releasing a SDXL model fine-tuned for the inpainting task?

[williamberman]

Yeah we plan on releasing an inpainting fine-tuned sdxl :) The current objective is more to have a working checkpoint than a reproducible training script

[abdellah-lamrani-alaoui]

That sounds great! @williamberman Do you have an ETA for releasing the model?

[williamberman]

@abdellah-lamrani-alaoui https://huggingface.co/diffusers/stable-diffusion-xl-1.0-inpainting-0.1 !

[GitHub1712]

Great to see this new inpaint model, could we train it ourself?

[GitHub1712]

@abdellah-lamrani-alaoui https://huggingface.co/diffusers/stable-diffusion-xl-1.0-inpainting-0.1 !

How did you made it? Is there any training code or example?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[lvZic]

How did you made it? Is there any training code or example?

How did you made it? Is there any training code or example?

[lawsonxwl]

@abdellah-lamrani-alaoui https://huggingface.co/diffusers/stable-diffusion-xl-1.0-inpainting-0.1 !

could you release the training code?

[crapthings]

Are there any scripts available for training today?

[sayakpaul]

https://github.com/huggingface/diffusers/pull/6922

[crapthings]

6922

This looks really good! Can it be used to train outpainting? For example, inverting the mask or masking out the entire background to keep only certain content.

[tingxueronghua]

6922

Could this reproduce similar performances?

[tingxueronghua]

6922

And is there any scripts for sd-xl-inpainting?

[sayakpaul]

Cc: @patil-suraj

[hzphzp]

same question