[Bug]: Soft Inpainting doesn't do anything: Automatic1111 Outperforms, Possible Forge Issue

[afrofail]

Checklist

• [X] The issue exists after disabling all extensions
• [X] The issue exists on a clean installation of webui
• [X] The issue is caused by an extension, but I believe it is caused by a bug in the webui
• [X] The issue exists in the current version of the webui
• [X] The issue has not been reported before recently
• [ ] The issue has been reported before but has not been fixed yet

What happened?

I recently came to an understanding while following a tutorial on inpainting that featured the "Soft Inpainting" option. Despite adhering to the instructions and utilizing the "Soft Inpainting" checkbox within Forge, it didn't perform as expected—especially when I increased the mask blur, which what was demonstrated in the video using the latest Automatic1111 fork. The inpainting results from Forge tended to be more blurred overall. In contrast, I've found that Automatic1111 provides much better outcomes, although Forge operates significantly faster and without VRAM errors. I would be immensely appreciative if this issue could be addressed. I applied the same "Soft Inpainting" settings on both platforms.

Steps to reproduce the problem

What should have happened?

When enabling "Soft Inpainting" it should preserve the masked selection and inpainting.

What browsers do you use to access the UI ?

Google Chrome

Sysinfo

sysinfo-2024-03-05-10-43.json

Console logs

None

Additional information

No response

[catboxanon]

Haven't tested to verify myself yet, but taking a cursory look at the infotext in the screenshots, you're using Euler a in both UIs, but in Forge, your eta noise seed delta (ENSD) value is set to 31337. Because you're using an ancestral sampler (the "a" part), that's very likely going to make a difference. If you use a non-ancestral sampler like Euler instead does the issue still occur?

[afrofail]

Haven't tested to verify myself yet, but taking a cursory look at the infotext in the screenshots, you're using Euler a in both UIs, but in Forge, your eta noise seed delta (ENSD) value is set to 31337. Because you're using an ancestral sampler (the "a" part), that's very likely going to make a difference. If you use a non-ancestral sampler like Euler instead does the issue still occur?

Thanks for noticing, unfortunately I get the same problem after changing the ENSD to -1. As you can see above, results are different, the "soft inpainting" feature in Automatic1111 seems more stable right now, especially when using SDXL models.

[catboxanon]

Could you upload the original image, mask, and generated images here for reproducibility purposes?

[afrofail]

[afrofail]

ENSD

Changed ENSD to 0 and still have the same issue between the two.

[catboxanon]

So it looks like this doesn't work as intended at all due to a large portion of the CFGDenoiser's forward function having been rewritten, and the relevant mask blending code being gutted out.

https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/eee46a5094f748ebe9fe4978b0d440a8ebbab4a6/modules/sd_samplers_cfg_denoiser.py#L168-L178

I think @lllyasviel needs to take a look at this one.

Edit: May have a fix soon -- wait warmly.

[catboxanon]

@afrofail Can you let me know if https://github.com/lllyasviel/stable-diffusion-webui-forge/pull/494 resolves this for you? It's not a complete solution as it undoes some changes by lvmin, but it would make this otherwise work.

[afrofail]

@afrofail Can you let me know if #494 resolves this for you? It's not a complete solution as it undoes some changes by lvmin, but it would make this otherwise work.

I just replied to the new commit merge but I'll post again here!

This fixed the issue, I can't thank you enough for figuring this out.

Forge

Automatic1111

[youlijie]

How do I apply it ? I changed the code manually like the commit but still do nothing for me.

[Sana-A-E]

The fix is supposedly already a part of the main branch but I can confirm that soft inpainting doesn't work as it should. In Forge, with identical settings (but really, the settings don't matter because it happens with different settings too while it never happens in Automatic1111 version), there are image artifacts on the edges where it should blend seamlessly. It is funny, but turning soft inpainting OFF produces a better inpaint.

Screenshot of strange artifacts I am talking about: (Note, though I am inpainting using a mask image to get the same results in all tests, the same thing happens even if you are doing the inpainting via inpainting tab) The area behind the armchair is masked and is supposed to be inpainted with something else.

This is what happens with the same seed/mask and all other settings with soft inpainting off:

And this is what Automatic1111 produces with same seed/mask/all other settings, with soft inpainting on:

Automatic1111 with soft inpainting off (surprisingly, results are nearly the same as with it on, perhaps it detects that used mask is not just black and white and automatically uses similar code to soft inpainting?):

(If it matters, the above examples are using Denoising 0.9 and latent noise fill, but the same happens with other values as well.)

I updated Forge just before posting this and these results are from the latest main branch commit. I noticed you closed some "soft inpainting is still not working" threads, but maybe they should be reopened.