search

lllyasviel / stable-diffusion-webui-forge

GNU Affero General Public License v3.0
7.76k stars 742 forks source link

[Bug]: ControlNet not working with Inpaint in "Inpaint not masked" mode #482

Open gshawn3 opened 7 months ago

gshawn3 commented 7 months ago

Checklist

What happened?

When using "Mask mode: Inpaint not masked", ControlNet does not work. This is inconsistent with the ControlNet implementation in Automatic1111/stable-diffusion-webui.

This issue is closely related to #76, which was fixed last month.

Steps to reproduce the problem

  1. Go to img2img/Inpaint
  2. Type a simple prompt and drop a picture onto the canvas
  3. Set Mask mode to "Inpaint not masked"
  4. Set Denoising Strength to 1
  5. Set ControlNet Unit 0 to Enable, Tile, tile_resample, control_v11f1e_sd15_tile, Control Weight 0.5
  6. Generate an image. Notice that the generated image has no ControlNet Tile processing applied to it. The generated image looks exactly the same as when ControlNet is disabled. Also notice that the ControlNet input preview is a completely black image.

Two more observations:

  1. When enabling "Upload independent control image" and using the same image, ControlNet works as expected.
  2. Switching the Mask mode to "Inpaint masked" and drawing a mask that covers the entire image works as expected. Again, the expectation is that "Inpaint not masked" with no mask is analogous to "Inpaint masked" with a full mask, and should result in the same behavior. This is how ControlNet works in stable-diffusion-webui.

What should have happened?

"Inpaint not masked" with no mask should behave the same as "Inpaint masked" with a full mask

What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome

Sysinfo

sysinfo-2024-03-04-11-12.json

Console logs

venv "C:/Users/Admin/Desktop/SD/stable-diffusion-webui/venv\Scripts\Python.exe"
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Version: f0.0.17v1.8.0rc-latest-273-gb9705c58
Commit hash: b9705c58f66c6fd2c4a0168b26c5cf1fa6c0dde3
Launching Web UI with arguments: --xformers --pin-shared-memory --cuda-malloc --cuda-stream --ckpt-dir C:/Users/Admin/Desktop/SD/stable-diffusion-webui/models/Stable-diffusion --hypernetwork-dir C:/Users/Admin/Desktop/SD/stable-diffusion-webui/models/hypernetworks --embeddings-dir C:/Users/Admin/Desktop/SD/stable-diffusion-webui/embeddings --lora-dir C:/Users/Admin/Desktop/SD/stable-diffusion-webui/models/Lora
Using cudaMallocAsync backend.
Total VRAM 24564 MB, total RAM 32530 MB
WARNING:xformers:A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
xformers version: 0.0.23.post1
Set vram state to: NORMAL_VRAM
Always pin shared GPU memory
Device: cuda:0 NVIDIA GeForce RTX 3090 Ti : cudaMallocAsync
VAE dtype: torch.bfloat16
CUDA Stream Activated:  True
2024-03-04 04:22:40.911261: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
WARNING:tensorflow:From C:\Users\Admin\Desktop\SD\stable-diffusion-webui\venv\lib\site-packages\keras\src\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

Using xformers cross attention
ControlNet preprocessor location: C:\Users\Admin\Desktop\SD\forge\models\ControlNetPreprocessor
Loading weights [3bb4fcd850] from C:\Users\Admin\Desktop\SD\stable-diffusion-webui\models\Stable-diffusion\sd_v1-5-pruned-emaonly.safetensors
2024-03-04 04:22:46,399 - ControlNet - INFO - ControlNet UI callback registered.
model_type EPS
UNet ADM Dimension 0
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 11.6s (prepare environment: 1.9s, import torch: 2.4s, import gradio: 0.6s, setup paths: 3.6s, other imports: 0.4s, load scripts: 1.4s, create ui: 1.0s, gradio launch: 0.1s).
Using xformers attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using xformers attention in VAE
extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale'}
left over keys: dict_keys(['alphas_cumprod', 'alphas_cumprod_prev', 'betas', 'log_one_minus_alphas_cumprod', 'model_ema.decay', 'model_ema.num_updates', 'posterior_log_variance_clipped', 'posterior_mean_coef1', 'posterior_mean_coef2', 'posterior_variance', 'sqrt_alphas_cumprod', 'sqrt_one_minus_alphas_cumprod', 'sqrt_recip_alphas_cumprod', 'sqrt_recipm1_alphas_cumprod'])
loaded straight to GPU
To load target model BaseModel
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) =  21591.593864440918
[Memory Management] Model Memory (MB) =  0.00762939453125
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  20567.586235046387
Moving model(s) has taken 0.01 seconds
To load target model SD1ClipModel
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) =  21590.358108520508
[Memory Management] Model Memory (MB) =  454.2076225280762
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  20112.15048599243
Moving model(s) has taken 0.05 seconds
Model loaded in 2.0s (load weights from disk: 0.1s, forge load real models: 1.3s, load VAE: 0.3s, calculate empty prompt: 0.2s).
2024-03-04 04:23:30,194 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE
2024-03-04 04:23:30,270 - ControlNet - INFO - Using preprocessor: tile_resample
2024-03-04 04:23:30,270 - ControlNet - INFO - preprocessor resolution = 0.5
Warning torch.load doesn't support weights_only on this pytorch version, loading unsafely.
[] []
2024-03-04 04:23:30,859 - ControlNet - INFO - Current ControlNet ControlNetPatcher: C:\Users\Admin\Desktop\SD\forge\models\ControlNet\control_v11f1e_sd15_tile.pth
To load target model AutoencoderKL
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) =  21217.16962814331
[Memory Management] Model Memory (MB) =  159.55708122253418
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  20033.612546920776
Moving model(s) has taken 0.03 seconds
2024-03-04 04:23:31,467 - ControlNet - INFO - ControlNet Method tile_resample patched.
To load target model BaseModel
To load target model ControlNet
Begin to load 2 models
Reuse 1 loaded models
[Memory Management] Current Free GPU Memory (MB) =  21084.882068634033
[Memory Management] Model Memory (MB) =  0.0
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  20060.882068634033
[Memory Management] Current Free GPU Memory (MB) =  21084.882068634033
[Memory Management] Model Memory (MB) =  689.0852355957031
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  19371.79683303833
Moving model(s) has taken 0.10 seconds
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 18/18 [00:14<00:00,  1.28it/s]

Additional information

No response

gshawn3 commented 7 months ago

Here are some screenshots to help illustrate the issue:

  1. Start by generating an image using txt2img, before sending it to img2img/inpaint: 1-txt2img

  2. Inpainting in "Inpaint not masked" mode. ControlNet is disabled for this gen: 2-nocontrolnet

  3. Now we've enabled ControlNet, but notice that it has no effect. The output is the same as image 2, when ControlNet was disabled: 3-controlnet-noimage

  4. After enabling "Upload independent control image" in the unit settings and uploading the initial image, you can see that ControlNet now works as expected. 4-controlnet-image

The expectation is that output image 3 is the same as image 4, because that is how ControlNet works in automatic1111/stable-diffusion-webui.