[Bug]: Inpainting mask does not adhere to correct boundaries

[bparrott78]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

When using inpainting->only masked, instead of using the precise mask, a rectangle is drawn at the resolution requested, but everything within the rectangle is denoised and replaced.

You can see from the above that I've very deliberately drawn an oddly shaped mask but the result shows that a rectangle around the mask is affected.

Steps to reproduce the problem

1. Use inpainting mask
2. Draw mask
3. hit generate
4. ???
5. see that a rectangle the that covers the x-y bounds of the mask has been affected

What should have happened?

The mask should precisely mark the edges of area affected.

Version or Commit where the problem happens

1.4.0.

What Python version are you running on ?

Python 3.10.x

What platforms do you use to access the UI ?

Windows

What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

Cross attention optimization

sdp-no-mem

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--ckpt-dir 'F:\\models\\stable-diffusion' --embeddings-dir 'F:\\embeddings' --vae-dir 'F:\\models\\VAE' --listen --enable-insecure-extension-access

List of extensions

Issue exists with all extensions turned off.

Console logs

venv "F:\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Version: v1.4.0
Commit hash: 394ffa7b0a7fff3ec484bcd084e673a8b301ccc8
Installing requirements

Checking diffusers version in requriments.txt
Correct diffusers version in requriments.txt
Current diffusers version: 0.17.1
diffusers is already up to date.

Check roop requirements
Install insightface==0.7.3
Installing sd-webui-roop requirement: insightface==0.7.3
Install onnx==1.14.0
Installing sd-webui-roop requirement: onnx==1.14.0
Install onnxruntime==1.15.0
Installing sd-webui-roop requirement: onnxruntime==1.15.0
Install tensorflow==2.12.0
Installing sd-webui-roop requirement: tensorflow==2.12.0
Install opencv-python==4.7.0.72
Installing sd-webui-roop requirement: opencv-python==4.7.0.72
Install diffusers==0.17.1
Installing sd-webui-roop requirement: diffusers==0.17.1

Installing requirements for TemporalKit extension

Launching Web UI with arguments: --ckpt-dir F:\\models\\stable-diffusion --embeddings-dir F:\\embeddings --vae-dir F:\\models\\VAE --listen --enable-insecure-extension-access
No module 'xformers'. Proceeding without it.
[-] ADetailer initialized. version: 23.6.2, num models: 8
[AddNet] Updating model hashes...
100%|█████████████████████████████████████████████████████████████████████████████| 111/111 [00:00<00:00, 20156.19it/s]
[AddNet] Updating model hashes...
100%|█████████████████████████████████████████████████████████████████████████████| 111/111 [00:00<00:00, 24155.22it/s]
2023-06-27 12:40:56,276 - ControlNet - INFO - ControlNet v1.1.224
ControlNet preprocessor location: F:\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
2023-06-27 12:40:56,393 - ControlNet - INFO - ControlNet v1.1.224
Loading weights [eafac48936] from F:\\models\\stable-diffusion\pb_finetune_and_notphoto.ckpt
Creating model from config: F:\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Couldn't find VAE named vae-ft-mse-840000-ema-pruned.ckpt; using None instead
Textual inversion embeddings loaded(41): 7dirtywords, Asian-Less-Neg, bad-artist, bad-hands-5, bad-picture-chill-1v, bad-picture-chill-32v, bad-picture-chill-75v, bad_prompt, corneo_runny_makeup, CyberRealistic_Negative-neg, easynegative, ng_deepnegative_v1_75t, rz-neg-15-foranalog, style-widow, Unspeakable-Horrors-16v, Unspeakable-Horrors-Composition-4v, verybadimagenegative_v1.3, vnappi_2, vnappi_2-000001, vnappi_2_2, was-mecha, xzfv-woman
Textual inversion embeddings skipped(16): CandyPunk, ChemPunk-96, InkPunk768, InkPunkHeavy768, InkPunkLandscapes768, InkPunkLite768, kanliu, Kipaki-125, Kipaki-150, nartfixer, nfixer, nrealfixer, PhotoHelper, PortraitHelper, rz-neg-general, VintageHelper
Model loaded in 3.3s (load weights from disk: 0.7s, create model: 0.3s, apply weights to model: 0.3s, apply half(): 0.5s, move model to device: 0.7s, calculate empty prompt: 0.7s).
2023-06-27 12:41:00,892 - roop - INFO - roop v0.0.1
2023-06-27 12:41:00,893 - roop - INFO - roop v0.0.1
Image Browser: ImageReward is not installed, cannot be used.
Applying attention optimization: sdp... done.
preload_extensions_git_metadata for 18 extensions took 0.87s
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 19.2s (import torch: 3.2s, import gradio: 0.8s, import ldm: 0.4s, other imports: 0.9s, list SD models: 0.1s, load scripts: 8.6s, create ui: 1.0s, gradio launch: 4.3s).

Additional information

No response

[bparrott78]

On further testing I can confirm this does not happen the first time the mask is drawn, but if you send the result back to inpaint, the original mask persists no matter how many times you attempt to clear it. However, if you press the 'x' to close the picture then send to inpaint, the correct mask is applied.

[chrme]

Increase mask blur, default = 4 Only masked padding = 32

[bparrott78]

Increase mask blur, default = 4 Only masked padding = 32

Does nothing.

[hellomynameisleo]

Automatic1111 has a bug where if you don't press the x button it will not clear the previous maskings even though you can't see it. You can test this out yourself by applying a mask, then sending another image for inpainting and them generate, it will use the previous mask along with the current mask if you don't press x.

[bparrott78]

It's not that. The issue is that the mask itself doesn't adhere to the boundaries. Typically you'll get a zoomed in picture of where the masking is being applied - the problem is that the masking doesn't apply to that zoomed in picture. It denoises and inpaints the whole area instead of just the masked part.

[leucome]

It's not that. The issue is that the mask itself doesn't adhere to the boundaries. Typically you'll get a zoomed in picture of where the masking is being applied - the problem is that the masking doesn't apply to that zoomed in picture. It denoises and inpaints the whole area instead of just the masked part.

I have the same issue trying to inpaint a hand and it process the whole image instead of just the hand, I have no idea what is causing this and it was working earlier this day. I dont know if something automatically updated or whatever but inpainting Only masked part is not working anymore. So I have to process the whole image. Let be honest on 6000PX image it is not something I want to do.

So far I restarted webui a couple of time refreshed the interface rebooted ect.. I just do not revert back to a working state. And I have no idea how to diagnose or report what is going on as there is no error message or log or anything. It just process the whole image and dont care about the "Only masked" option.

There is other report of the same or a similar bug. But I am not sure it is 100% it is identical. As this in this case the mask is working while some other case have the mask not working. So this is specifically about not cropping the image before processing it.

[mjdav45]

I've found this happens repeatably if you're using the upload mask feature and you've resized the mask image before uploading even if it doesn't affect the mask drawing itself (e.x. expanding the canvas so it matches the base image resolution then resaving). Even if it looks correct visually, there's something about how the mask is interpreted that causes large rectangular blocks around the mask to also be inpainted.

The easiest fix when this happens is to recreate the mask image in the matching base resolution and do not resize the canvas afterwards.

It also seems to bug out similarly if your mask covers multiple separate areas of the image or touches too much of the edge of the image.

[creeduk]

Not sure if this is related but I found masks touching or close to the edge seem to bleed. The best way to see what is happening is to enable "For inpainting, save a copy of the greyscale mask" you get an image of the mask and you can see extra mask elements. For me if it touches the edge it usually makes a rectangle running along that whole edge. Even close to the edge masks like a big circle. I had some success with upload mask but again if that mask is on the edge it bleeds. Usually if I am using inpainting to increase the canvas size.