Open Svyaga opened 2 months ago
Forgot to mention that there was a similar problem in ComfyUI and it was fixed later: https://github.com/comfyanonymous/ComfyUI/issues/1363
And here's how it was fixed: https://github.com/comfyanonymous/ComfyUI/commit/bed116a1f916f23b65c7b68cf60d09b2401fcf29
Seems to be the same problem as in ComfyUI, here are some examples of misaligned pixels in initial generation of 888x1184 image:
I also should add that this happens on all sides of the borders, but it's more visible on bottom and right borders, especially right one.
The first example does look like an issue I've seen when denoising padded latents, which may be the case if your region dimensions are not multiples of 64. Reflect padding instead of constants usually worked best in my experiments.
Your second example confuses me, because pixels far outside of your mask area are being affected. That looks like VAE nondeterminism to me, but the canvas paste-back operation shouldn't allow that. What is your Mask Blur setting at? Seems like it might be way too high.
The first example does look like an issue I've seen when denoising padded latents, which may be the case if your region dimensions are not multiples of 64. Reflect padding instead of constants usually worked best in my experiments.
Your second example confuses me, because pixels far outside of your mask area are being affected. That looks like VAE nondeterminism to me, but the canvas paste-back operation shouldn't allow that. What is your Mask Blur setting at? Seems like it might be way too high.
Here's 1024x1024 image without any img2img applied:
You can notice the misaligned pixels there:
Info by dunkeroni: https://discord.com/channels/1020123559063990373/1240001897814036562
Is there an existing issue for this problem?
Operating system
Windows
GPU vendor
Nvidia (CUDA)
GPU model
RTX 3070
GPU VRAM
8GB
Version number
4.2.0
Browser
Chrome 124.0.6367.158
Python dependencies
{ "accelerate": "0.29.2", "compel": "2.0.2", "cuda": "12.1", "diffusers": "0.27.2", "numpy": "1.26.4", "opencv": "4.9.0.80", "onnx": "1.15.0", "pillow": "10.2.0", "python": "3.10.11", "torch": "2.2.2+cu121", "torchvision": "0.17.2+cu121", "transformers": "4.39.3", "xformers": "0.0.25.post1" }
What happened
SD: SDXL. Whenever image is generated during img2img/inpaint process, strange artifacts appear along the borders of the image. Tested on both Tiled img2img (in "Workflows" mode) and in Canvas mode.
You can see a thin artifact line along the border of the image (top arrow), it was caused by tiled img2img:
Then I made a slight inpaint (bottom arrow) and tried to inpant. New artifacts along the border appeared:
![3](https://github.com/invoke-ai/InvokeAI/assets/143509676/6d1f3be5-e817-42d8-9398-3d9d1420b989)
These artifacts appear even if masked area (for inpaint) is far away from the border. This is the masked area:
This is before/after inpaint. You can notice that border of the image was changed: https://imgsli.com/MjYyOTkw
What you expected to happen
I expected selected areas to be properly cropped and inpaint to be properly applied without artifacts along borders of the image.
How to reproduce the problem
These steps copy my actions, but there are probably some optional steps like upscaling an image. I will mark them with "*".
Tiled img2img: 1.1. Generate any image. 1.2.* Upscale image with x4 upscaler (I used RealESRGAN x4 Plus (anime 6B) 1.3. Run tiled img2img workflow: Tiled img2img.json
Canvas inpaint 2.1. Generate any image 2.2. Upscale image with x4 upscaler (I used RealESRGAN x4 Plus (anime 6B) 2.3. Put upscaled image in Canvas mode. 2.4. Select any area of the image, any of the borders of selection box should align with image border. 2.5. Mask any area inside the selection box 2.6. Run inpaint.
Additional context
I noticed that these artifacts appear along the borders during preview of the generation:![1](https://github.com/invoke-ai/InvokeAI/assets/143509676/8acd7236-2fcc-4c68-ae74-9f7655d56f61)
Discord username
svyaga