search

Nerogar / OneTrainer

OneTrainer is a one-stop solution for all your stable diffusion training needs.
GNU Affero General Public License v3.0
1.81k stars 154 forks source link

[Feat]: Train flux inpainting #506

Open thesby opened 1 month ago

thesby commented 1 month ago

Describe your use-case.

How to train flux inpainting model? I have not find the preset.

What would you like to see as a solution?

A preset for flux inpainting training config

Have you considered alternatives? List them here.

I have not find alternatives?

Nerogar commented 1 month ago

Is there a specific inpainting model you have in mind? As far as I know, there is no official Flux inpainting model

thesby commented 1 month ago

https://huggingface.co/spaces/SkalskiP/FLUX.1-inpaint

thesby commented 1 month ago

I tried to use comfyui flux inpainting workflow. Sometimes the boundary is bad even though the mask is blurred.

Arcitec commented 1 week ago

FLUX doesn't need inpainting models and officially doesn't have any. You need specific ComfyUI workflows though. Can't remember the details but basically you need some sampler control attention mask or something like that. There's tons of tutorials about it.

The general idea is: You generate noise in the areas you want to inpaint, but nowhere else. Then, it only applies denoising (sampling) to those areas, smoothly blending between the original image and the inpaint area. And since the original pixel data is still available, FLUX will always be aware of the surrounding context and will create good blending.

If you don't do this, and you only use some basic img2img and mask/fade, then you will not get good boundaries. You need masking to happen at the sampling stage! And you should not be using any kind of img2img process!

Nerogar commented 1 week ago

@Arcitec That method works for all models, not just Flux. But the results are always worse than specifically trained inpainting models. for SD1.5 and SDXL the inpainting models have special mask and conditioning inputs that help the model understand which parts of the image to inpaint. It's a bit unfortunate that nobody bothers training them anymore. Kolors is another slightly more modern model with a dedicated inpainting model, but I didn't have time to add support yet.

Arcitec commented 1 week ago

Ahh I see. Well, good news!

Flux just came out with an inpainting model.

https://blackforestlabs.ai/flux-1-tools/