Feature Request: Training Lora on inpainting models.

[miniwa]

As a developer, one of the challenges I've encountered is that lora networks, when trained on standard models, tends to underperform when used with inpainting models.

The possible solution: Allowing creators to publish two variants of a Lora for the same concept - a "normal" Lora and an "inpainting" Lora. This would empower users with the flexibility to choose the best-suited model for their specific applications, thus improving overall user experience and satisfaction.

cloneofsimo has implemented this in his repo. Probably can use the same method as he does. Some examples below:

• If training inpainting model, also generate random mask: https://github.com/cloneofsimo/lora/blob/bdd51b04c49fa90a88919a19850ec3b4cf3c5ecd/lora_diffusion/dataset.py#L268
• Function to generate random mask: https://github.com/cloneofsimo/lora/blob/bdd51b04c49fa90a88919a19850ec3b4cf3c5ecd/lora_diffusion/dataset.py#108
• Use dataset to generate batch: https://github.com/cloneofsimo/lora/blob/bdd51b04c49fa90a88919a19850ec3b4cf3c5ecd/lora_diffusion/cli_lora_pti.py#L198
• Simple extra argument to training script: https://github.com/cloneofsimo/lora/blob/master/training_scripts/inpainting_example.sh

Thank you for all your hard work!

[ceswap]

Yes, that would be really nice to have +1

[Vadmidin]

Yes, please implement!

[kohya-ss]

Thank you for bringing up this issue. The idea of training LoRA for inpainting models is intriguing indeed. Also, I appreciate your reference to cloneofsimo's repository.

However, there are several challenges to consider in the implementation.

Firstly, our dataset would need to be adjusted to handle additional information beyond images, such as masks. While cloneofsimo's code could provide significant guidance, we also need to consider Aspect Ratio Bucketing, which could be time-consuming.

Another issue lies in the inference environment. This repository does not currently contain a script for inference using inpainting models. Therefore, to generate images combining LoRA, we would need to create a new generation script. Moreover, I'm not sure Automatic1111's Web UI supports LoRA with inpainting models, and also additional-network extension needs to be updated.

Regarding the first issue, similar functionality is also required for ControlNet and mask loss. Thus, I plan to implement this in the near future. Once that is done, I hope to further consider this issue.

I appreciate your understanding on this matter.

[miniwa]

Regarding the first issue, similar functionality is also required for ControlNet and mask loss. Thus, I plan to implement this in the near future. Once that is done, I hope to further consider this issue.

These are great features on their own. :)

[HG2407]

any updates ?

[julien-blanchon]

Any news ?

[PMykhailo]

How do you even train lora for img2img? I searched for methods, but find none. I'm interesting particularly in inpainting

[PMykhailo]

Maybe this will help some of you all. Nut tested it myself, yet https://github.com/huggingface/diffusers/tree/main/examples/research_projects/dreambooth_inpaint