Inpaint changes parts of original image

[worldemar]

Not sure if it is an issue that could or should be solved. Inpaint seem to be using image mask to draw-in parts of image that are transparent. Despite that it changes already painted (non-transparent) parts of image too.

How to reproduce:

• fill entire image with one color
• erase a hole that is smaller than 512x512
• increase selection size to bigger than transparent hole
• position selection in such a way that it covers entire hole and has all edges on solid color
• use Inpaint

Expected result:

• transparent hole is filled with whatever Diffusion has produced
• parts that were already painted (solid color) are not changed

Observed result:

• transparent hole is filled with whatever Diffusion has produced
• solid color parts have very noticeable selection edge etched into it
• solid color seem to be changed to slightly different hue

While using on regular images repeated Inpaint-s that cover same area seem to change contrast/saturation of previously painted parts. How to reproduce:

• resize selection to something small, for example 64x64
• generate something at center of image
• resize selection to larger size, for example 128x128
• inpaint several times while covering entire original 64x64 square

Expected result:

• transparent parts of image are painted
• original 64x64 square is not changed

Observed result:

• over several iterations of Inpaint original square becomes more and more oversharpened and over-saturated

Another easy way to get this result is to Generate anything and then Inpaint many times without changing selection. Doing Inpaint 10 times or so will turn almost any result into colorful noise.

As far as I was able to debug both of those are manifestations of same issue, maybe something to do with masking.

[ahrm]

Yes, I have noticed this issue. I am pretty sure it is a stable diffusion bug though so I can't easily fix it, because we don't do anything fancy to the images, we pass them verbatim to huggingface's stable diffusion inpainting function.

[worldemar]

Yes, I have noticed this issue. I am pretty sure it is a stable diffusion bug though so I can't easily fix it, because we don't do anything fancy to the images, we pass them verbatim to huggingface's stable diffusion inpainting function.

From reading the code I assumed (probably, incorrectly) that after stable diffusion inpainting function had returned the "inpaint image part" it is also multiplied by mask, leaving only transparent-in-original-image pixels in it so that we do not overwrite anything

[ahrm]

No we don't do that. We use the entire image that inpainting function returns. I could try replacing just the inpainted part, but I assume we will have the same issue but instead of being in the square it is just in the inpainted part. I will try it out and implement it if the results are better in that case.

[ahrm]

I tried it out and as expected, the results are much worse when we only apply the inpainted part. It appears that stable diffusion modifies the surrounding part a little bit to make a more natural image.

If you want to try it out, you just have to change this line to this:

# add mask as alpha channel
inpainted_image = np.concatenate([inpainted_image, mask[:, :, np.newaxis]], axis=2)
self.set_selection_image(inpainted_image)

[ahrm]

Added the ability to use a smoothed version of the mask instead of the whole inpainted image in 76eaa6cf484c8520f3313b1074711de92e0a00a1. The results are much better now:

original image:

without smoothing:

with smoothing:

(need to squickly swap between images to see the difference)