[Bug]: 2 Issues with Controlnet and Inpainting

TheRealPhooney commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

1) Mask is ignored when Controlnets are enabled without one of the enabled Controlnets being the Inpainting one. In this case, it ignores the drawn mask and behaves as img2img.

2) Inpainting with only a small area masked results in only that small area being the output as if the image has zoomed in on that one area. The larger the area you mask, the larger the area is returned in the output. If you put a small dot for the mask in each corner of the original image you can retain the original content.

Steps to reproduce the problem

1) Example: Openpose and Depth are enabled, no other controlnets = mask ignored, entire image altered. Example: Openpose and Inpainting are enabled = mask respected and changes (mostly) only happen within the masked area, though there does appear to be some changes applied to the entire image, most noticably colors. I guess this is part of the process to help blend the changes inside the masked area?

2) Example: a 512x512 image full body of a man that covers most of the available space. Mask over his head, set the output to 512x512. Have controlnet(s) enabled (I tested with openpose, canny, depth zoe and inpainting), and the output image will be a 512x512 image of just the man's head and the area immediately around it.

Example: a 512x512 image full body of a man that covers most of the available space. Mask over top left quadrant, set output to 512x512. Have controlnet(s) enabled (I tested with openpose, canny, depth zoe and inpainting), and the output image will be a 512x512 image with the content from the top left quadrant and some area around it.

Example: a 512x512 image full body of a man that covers most of the available space. Mask over anything and also place masking dots in each corner. Set the output to 512x512, have controlnet(s) enabled and output should frame the same area.

Tested with all resize modes and pixel perfect on and off.

What should have happened?

1) Drawn/uploaded mask should be respected when using controlnet(s) even if not using controlnet inpainting.

2) Output should not be zoomed in on the masked area, the output image should frame the same area as the original image.

Commit where the problem happens

webui: 22bcc7be428c94e9408f589966c2040187245d81 controlnet: v1.1.125

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--xformers --api

Console logs

Not full logs:

Loading preprocessor: openpose_full
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.RESIZE
raw_H = 1080
raw_W = 1920
target_H = 1080
target_W = 1920
estimation = 1080.0
preprocessor resolution = 1088
Loading model: control_v11f1p_sd15_depth_fp16 [4b72d323]
Loaded state_dict from [C:\***\StableDiffusion\stable-diffusion-webui-master2\webui\extensions\sd-webui-controlnet\models\control_v11f1p_sd15_depth_fp16.safetensors]
Loading config: C:\***\StableDiffusion\stable-diffusion-webui-master2\webui\extensions\sd-webui-controlnet\models\control_v11f1p_sd15_depth.yaml
ControlNet model control_v11f1p_sd15_depth_fp16 [4b72d323] loaded.
Loading preprocessor: depth_zoe
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.RESIZE
raw_H = 1080
raw_W = 1920
target_H = 1080
target_W = 1920
estimation = 1080.0
preprocessor resolution = 1088
Loading model from cache: control_v11p_sd15_inpaint_fp16 [be8bc0ed]
using inpaint as input
Loading preprocessor: inpaint
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.RESIZE
raw_H = 1080
raw_W = 1920
target_H = 1080
target_W = 1920
estimation = 1080.0
preprocessor resolution = 1088
100%|██████████████████████████████████████████████████████████████████████████████████| 50/50 [01:34<00:00,  1.90s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 50/50 [01:35<00:00,  1.92s/it]
Loading model: control_v11p_sd15_openpose_fp16 [73c2b67d]██████████████████████████████| 50/50 [01:35<00:00,  1.88s/it]

Additional information

No response

lllyasviel commented 1 year ago

are you in img2img with " inpaint only masked" selected?

TheRealPhooney commented 1 year ago

are you in img2img with " inpaint only masked" selected?

In img2img > Inpaint and yes inpaint only masked is selected

TheRealPhooney commented 1 year ago

Just tested it with "whole picture" selected instead.

This causes it to render the expected area rather than zooming in, but it ignores the drawn mask even with controlnet inpaint enabled.

TheRealPhooney commented 1 year ago

Hold that thought, I just closed the console and the browser and restarted it, now "whole picture" does not cause the mask to be ignored.

edit: (and it renders the expected area of the picture and does not zoom in)

lllyasviel commented 1 year ago

I am going to close this, feel free to ask more specific questions so that it can be opened again

TheRealPhooney commented 1 year ago

Sorry... are the issues fixed?

Or, for the time being at least, for issue (1) is it the intended behaviour that if we have any controlnets enabled, we always have to have one of the controlnets be the inpainting preprocessor/model in order for the drawn/uploaded mask to be used?

and for issue (2) is it the intended behaviour that we cannot use inpaint area "only masked" and we must use "whole picture" so that the rendered image is not zoomed in?

lllyasviel commented 1 year ago

hello we cannot understand the description. can you provide specific screenshots of errors and descriptions of your expected results?

TheRealPhooney commented 1 year ago

OK... I'll try to illustrate with screenshots.

Here is an original image, (the same image I put in the canny controlnet below) 512x512:

seaserpent2

Here is the image with mask drawn on:

mask drawn

Prompt is "a top hat"

Expected result (achieved with no controlnets) something like this:

Expected result

Actual result (using settings whole picture, controlnet canny preprocessor/model, but NO inpainting controlnet preprocessor/model):

whole picture-canny-noCNinpaint

Actual result (Using settings only masked, controlnet canny preprocessor/model, but NO inpainting controlnet preprocessor/model):

maskedonly-canny-noCNinpaint

As you can see, without using a controlnet inpainting preprocessor/model the drawn mask is ignored, the whole image is altered as if it was in img2img not inpaint, and when only masked is selected, the rendered image is zoomed in on the masked area.

Now adding a second controlnet, for inpainting preprocessor/model and using only masked:

only masked-canny-yesCNinpaint

Again, the rendered image is zoomed in on the masked area. This last one is not a great example... for whatever reason I could not get it to put a top hat on the sea serpent after several tries, but the masked area is not the only area altered in the generation.

lllyasviel commented 1 year ago

cannot reproduce the problem

TheRealPhooney commented 1 year ago

Anything you can see in my controlnet settings that needs to change?

lllyasviel commented 1 year ago

i think the cofig is ok. feel free to give specific problems and we will reopen issue.

TheRealPhooney commented 1 year ago

Sorry... I'm not sure how much more specific you'd like me to be?

lllyasviel commented 1 year ago

it seems that mikubill's code "Skip img2img processing when using img2img initial image" is causing problems, we gonna to remove this dangerous option

TheRealPhooney commented 1 year ago

Unticking that option does seem to solve the problem... kinda busy just this minute, but I'll double check everything over the next day or so (unless you update and remove it in the mean time, in which case... all good!)

kronkinatorix commented 1 year ago

I'm having a similar issue - inpainting is not working.

To be specific about the problem, I'm unable to use lineart with inpainting. This is the case with the inpaining_harmonious preprocessor on / off, and with the inpainting model on/off, and every combination therein.

When inpainting starts my console prints "preprocessor resolution = 64" whether I have a preprocessor for the lineart module on or not. - changing the resolutiion on the preprocessor does not change this. Creating a preview of the processed image, downloading it, uploading it as the image to use, and turning off the preprocessor does not change anything.

Controlnet appears to be trying to use a part of the image rather than the whole image for inpainting.

Here is the processed image I'm trying to use;

https://imgur.com/a/GqR8ugc

Here is the preview that Controlnet creates when inpainting:

The results range blurriness: https://imgur.com/a/yxoHqaE
(A color photo of charlie Chaplin Steps: 35, Sampler: DPM++ 2M Karras, CFG scale: 1, Seed: 3716622201, Size: 768x768, Model hash: 52c8653a97, Model: a1-inpainting_Realistic_Vision_V1.4-inpainting, Denoising strength: 1, Conditional mask weight: 1.0, Mask blur: 2, ControlNet 0 Enabled: True, ControlNet 0 Preprocessor: lineart_standard (from white bg & black line), ControlNet 0 Model: control_v11p_sd15_lineart [43d4be0d], ControlNet 0 Weight: 2, ControlNet 0 Starting Step: 0, ControlNet 0 Ending Step: 1, ControlNet 0 Resize Mode: Crop and Resize, ControlNet 0 Pixel Perfect: False, ControlNet 0 Control Mode: Balanced, ControlNet 0 Preprocessor Parameters: "(1024, 64, 64)", ControlNet 1 Enabled: True, ControlNet 1 Preprocessor: inpaint_global_harmonious, ControlNet 1 Model: control_v11p_sd15_inpaint [ebff9138], ControlNet 1 Weight: 1, ControlNet 1 Starting Step: 0, ControlNet 1 Ending Step: 1, ControlNet 1 Resize Mode: Crop and Resize, ControlNet 1 Pixel Perfect: False, ControlNet 1 Control Mode: Balanced, ControlNet 1 Preprocessor Parameters: "(64, 64, 64)")

to making no changes to the image at all: https://imgur.com/a/29gaI0O Steps: 35, Sampler: DPM++ 2M Karras, CFG scale: 1, Seed: 2854610815, Size: 768x768, Model hash: 52c8653a97, Model: a1-inpainting_Realistic_Vision_V1.4-inpainting, Denoising strength: 0, Conditional mask weight: 1.0, Mask blur: 2, ControlNet 0 Enabled: True, ControlNet 0 Preprocessor: lineart_standard (from white bg & black line), ControlNet 0 Model: control_v11p_sd15_lineart [43d4be0d], ControlNet 0 Weight: 2, ControlNet 0 Starting Step: 0, ControlNet 0 Ending Step: 1, ControlNet 0 Resize Mode: Crop and Resize, ControlNet 0 Pixel Perfect: False, ControlNet 0 Control Mode: Balanced, ControlNet 0 Preprocessor Parameters: "(1024, 64, 64)", ControlNet 1 Enabled: True, ControlNet 1 Preprocessor: inpaint_global_harmonious, ControlNet 1 Model: control_v11p_sd15_inpaint [ebff9138], ControlNet 1 Weight: 1, ControlNet 1 Starting Step: 0, ControlNet 1 Ending Step: 1, ControlNet 1 Resize Mode: Crop and Resize, ControlNet 1 Pixel Perfect: False, ControlNet 1 Control Mode: Balanced, ControlNet 1 Preprocessor Parameters: "(64, 64, 64)" Template: A color photo of charlie Chaplin)88

The Skip img2image initial image option is off - I'm using hte latest version of controlnet/automatic11118

Mikubill / sd-webui-controlnet