[Bug]: Inpaint training conditional image must be unmasked

[GitHub1712]

What happened?

Training inpainting uses GenerateMaskedConditioningImage generating the conditioning image with the inpainting mask. The inpainting training typically works like this:

1. The model is given the masked image and the mask.
2. It's also given the original unmasked image as conditioning.
3. The model learns to predict the pixels in the masked area, aiming to match the original image.

By using the unmasked image as conditioning, the model can learn to generate content that's consistent with the overall image, even for large masked areas.

• confusing config options to train inpainting masked and unmasked, inpainting training always needs to be masked but not the conditioning image and not additional to the inpainting mask.

What did you expect would happen?

• conditioning unmasked to learn to recunstruct masked
• more clear config or documentation about inpaint training
• correct me if I am wrong

Relevant log output

No response

Output of pip freeze

No response

[Nerogar]

I'm not sure what you are trying to say. The conditioning image is the image passed into the model, alongside the mask. It has nothing to do with the training objective.

inpainting training always needs to be masked

no it doesn't. You can train an inpainting model without masks, meaning a completely blacked out conditioning image.

... not additional to the inpainting mask

You can either add a manual mask, or automatically generate a mask during training using the enable_random_circular_mask_shrink option of a concept.

[GitHub1712]

I'm not sure what you are trying to say. The conditioning image is the image passed into the model, alongside the mask. It has nothing to do with the training objective.

Okay, thank you, I was confused thinking conditioning image is the training objective. Maybe a better name would be masked_input_image

no it doesn't. You can train an inpainting model without masks, meaning a completely blacked out conditioning image.

Yes but without masks it would not train the inpaint ability and move model back into text2image direction. Still confused, preset "#sd 1.5 inpaint" uses inpaint masks for inpaint training and "#sd 1.5 inpaint masked" uses additional masks to not train masked parts? Or "#sd 1.5 inpaint" uses no inpaint masks and "#sd 1.5 inpaint masked" is what I need to train normal inpainting? If I want to improve inpaint abilitys on my images (textures) and own kind of masks (-masklabel.png´s).

So the bug was a misunderstanding, inpaint training documentation would be great. Maybe we could include a small working inpaint training exampl dataset with masks?

[Nerogar]

So the bug was a misunderstanding, inpaint training documentation would be great. Maybe we could include a small working inpaint training exampl dataset with masks?

Very few people train inpainting models, that's why there isn't much documentation. The general idea is pretty simple though. In the concept window, you can see the mask that's used for training. The image will be lighter inside the masked area. You can either create a manual mask, or let OneTrainer generate a circular mask for you by enabling the "Circular Mask Generation" switch.

About the lack of documentation, feel free to add more information to the wiki once you figure everything out.

[GitHub1712]

Okay, thank you very much! I believe finetuning inpainting is very interesting for some, because there is no inpaint training example for SD inpaint models at all in any open source project. I would like to create a simple inpaint training doc/example if I would understand better.

My main question: Does inpaint 15 training preset without mask in preset name use inpaint masks if there are -masklabel.png´s or "Circular Mask Generation" active and "Maksed Training" NOT active in trainig tab?

It seems so, meanwhile from my experience and code review/debugging. I have a 7600 texture dataset with -masklabel.pngs and tried all kind of settings to finetune 1.5 inpainting. Masked training gives bad results, the model begins to fill with masks or dark very quick. Training without masks seems to be the inpaint training I need. It trains well generation of those textures but not improves the inpainting as I hope, inpainted areas are still a little different, similar to without finetuning on those textures. I keep trying. Also the VAE has a huge effect for the inpainting of texure details but finetuning the VAE with the inpainting breaks the VAE quickly.

[Nerogar]

Does inpaint 15 training preset without mask in preset name use inpaint masks if there are -masklabel.png´s or "Circular Mask Generation" active and "Maksed Training" NOT active in trainig tab?

No. if "Masked Training" is disabled on the training tab, no masks will be used during training. It will essentially train the model as a normal model, not as an inpainting model. You can play around with the "unmasked probability" value. I believe the original SD1.5 inpainting model was trained with this setting set to 0.25, which means 25% of training steps will be trained like a normal SD model

[GitHub1712]

That was fast, strange because of my masked training experience but I will try "unmasked probability". Could there be an issue in the logic, the conditioning image or training objective beeing masked? Cause it should learn to reconstruct the whole image with masked input but it learns to reconstruct masks for me, so it goes in the wrong direction.

[GitHub1712]

My debug conditioning images are looking like that (yes the mask is big on this): I believe the grey is from Unmasked weight 0.3 which does not explain rgb 124,124,123. Maybe unmasked weight should not make mask brighter but fade with the image like this?