search

Mikubill / sd-webui-controlnet

WebUI extension for ControlNet
GNU General Public License v3.0
17.04k stars 1.96k forks source link

[Bug]: Enabling and disabling controlnet slows down SD generation speed #1971

Closed DNS-Midnattsol closed 11 months ago

DNS-Midnattsol commented 1 year ago

Is there an existing issue for this?

What happened?

After enabling the ControlNet plugin for A1111, using it for an image generation, then disabling it, all subsequent image generations take ~5 times longer while providing the same image result.

Normal text-to-img generation speed: 1.24it/s enabling ControlNet: 4.17s/it disabling ControlNet again: 4.15s/it

Only restarting the WebUI currently fixes the issue.

I initially posted this report on A1111s Github, i was told this here would be the proper place instead.

System: I5-3570K, GTX 1060 6GB, 32GB Ram

Graphics driver: 31.0.15.3667 (536.67 ?)

A1111 WebUI v1.5.1, ControlNet v1.1.311

SD parameters used to confirm this issue after a fresh WebUI restart (any parameters should result in the issue, it seems to not be specific to them); initial image generation result: 01_InitialNormalSpeedGeneration

Enabling ControlNet with some preprocessor; result of the subsequent generation: 02_Enable_CN_Openpose

Disabling ControlNet and running the generation again, with identical settings to the initial image generation; generation result: 03_Disable_CN_and_generate

Steps to reproduce the problem

  1. Start the A1111 WebUI.
  2. Enter a positive prompt in text-to-img and generate a 512² image.
  3. Enable ControlNet, select a mode so preprocessors load.
  4. Hit 'Generate' again, wait until it is done.
  5. Disable ControlNet, select 'all' to remove the preprocessor selection, hit 'Generate' again.
  6. Compare generation speeds of the first and third image generation.

What should have happened?

Image generation speed should have returned to around 1.24it/s after disabling ControlNet.

Commit where the problem happens

ControlNet v1.1.311 A1111 webui v1.5.1  •  python: 3.10.6  •  torch: 2.0.1+cu118  •  xformers: 0.0.20  •  gradio: 3.32.0  •  checkpoint: 980cb713af

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

--xformers --medvram --no-half

List of enabled extensions

sd-webui-3d-open-pose-editor sd-webui-controlnet sd-webui-inpaint-anything sd-webui-openpose-editor

Console logs

Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.5.1
Commit hash: 68f336bd994bed5442ad95bad6b6ad5564a5409a

Launching Web UI with arguments: --xformers --medvram --no-half
2023-08-24 21:25:28,326 - ControlNet - INFO - ControlNet v1.1.311
ControlNet preprocessor location: E:\Automatic1111 WebUI\webui\extensions\sd-webui-controlnet\annotator\downloads
2023-08-24 21:25:28,530 - ControlNet - INFO - ControlNet v1.1.311
Loading weights [980cb713af] from E:\Automatic1111 WebUI\webui\models\Stable-diffusion\reliberate_v10.safetensors
Creating model from config: E:\Automatic1111 WebUI\webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 19.5s (launcher: 4.2s, import torch: 4.8s, import gradio: 1.3s, setup paths: 0.9s, other imports: 1.2s, setup codeformer: 0.1s, load scripts: 5.7s, create ui: 0.9s, gradio launch: 0.3s).
DiffusionWrapper has 859.52 M params.
Applying attention optimization: xformers... done.
Model loaded in 3.9s (load weights from disk: 1.0s, create model: 1.6s, apply weights to model: 0.9s, calculate empty prompt: 0.4s).
100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:21<00:00,  1.18it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 25/25 [00:20<00:00,  1.24it/s]
2023-08-24 21:29:45,244 - ControlNet - INFO - Loading model: control_v11p_sd15_openpose [cab727d4]:20<00:00,  1.33it/s]
2023-08-24 21:29:45,885 - ControlNet - INFO - Loaded state_dict from [E:\Automatic1111 WebUI\webui\extensions\sd-webui-controlnet\models\control_v11p_sd15_openpose.pth]
2023-08-24 21:29:45,885 - ControlNet - INFO - Loading config: E:\Automatic1111 WebUI\webui\extensions\sd-webui-controlnet\models\control_v11p_sd15_openpose.yaml
2023-08-24 21:29:49,202 - ControlNet - INFO - ControlNet model control_v11p_sd15_openpose [cab727d4] loaded.
*** Error running process: E:\Automatic1111 WebUI\webui\extensions\sd-webui-controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "E:\Automatic1111 WebUI\webui\modules\scripts.py", line 519, in process
        script.process(p, *script_args)
      File "E:\Automatic1111 WebUI\webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 741, in process
        input_image, image_from_a1111 = Script.choose_input_image(p, unit, idx)
      File "E:\Automatic1111 WebUI\webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 654, in choose_input_image
        raise ValueError('controlnet is enabled but no input image is given')
    ValueError: controlnet is enabled but no input image is given

---
100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [01:44<00:00,  4.18s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 25/25 [01:44<00:00,  4.17s/it]
100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [01:43<00:00,  4.15s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 25/25 [01:43<00:00,  4.15s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 25/25 [01:43<00:00,  4.09s/it]

Additional information

You can actually load an image into ControlNet or not, it doesn't change this issue.

So when loading in a pose as image, the Console logs change to the following:

Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] Version: v1.5.1 Commit hash: 68f336bd994bed5442ad95bad6b6ad5564a5409a

Launching Web UI with arguments: --xformers --medvram --no-half 2023-08-24 22:55:23,648 - ControlNet - INFO - ControlNet v1.1.311 ControlNet preprocessor location: E:\Automatic1111 WebUI\webui\extensions\sd-webui-controlnet\annotator\downloads 2023-08-24 22:55:23,839 - ControlNet - INFO - ControlNet v1.1.311 Loading weights [980cb713af] from E:\Automatic1111 WebUI\webui\models\Stable-diffusion\reliberate_v10.safetensors Creating model from config: E:\Automatic1111 WebUI\webui\configs\v1-inference.yaml LatentDiffusion: Running in eps-prediction mode Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch(). Startup time: 18.9s (launcher: 4.0s, import torch: 4.5s, import gradio: 1.3s, setup paths: 1.0s, other imports: 1.2s, setup codeformer: 0.1s, load scripts: 5.6s, create ui: 0.9s, gradio launch: 0.3s). DiffusionWrapper has 859.52 M params. Applying attention optimization: xformers... done. Model loaded in 4.0s (load weights from disk: 1.0s, create model: 1.4s, apply weights to model: 1.1s, calculate empty prompt: 0.4s). 100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:22<00:00, 1.11it/s] Total progress: 100%|██████████████████████████████████████████████████████████████████| 25/25 [00:21<00:00, 1.14it/s] 2023-08-24 22:56:24,070 - ControlNet - INFO - Loading model: control_v11p_sd15_openpose [cab727d4]:21<00:00, 1.25it/s] 2023-08-24 22:56:24,765 - ControlNet - INFO - Loaded state_dict from [E:\Automatic1111 WebUI\webui\extensions\sd-webui-controlnet\models\control_v11p_sd15_openpose.pth] 2023-08-24 22:56:24,765 - ControlNet - INFO - Loading config: E:\Automatic1111 WebUI\webui\extensions\sd-webui-controlnet\models\control_v11p_sd15_openpose.yaml 2023-08-24 22:56:28,210 - ControlNet - INFO - ControlNet model control_v11p_sd15_openpose [cab727d4] loaded. 2023-08-24 22:56:28,320 - ControlNet - INFO - Loading preprocessor: none 2023-08-24 22:56:28,320 - ControlNet - INFO - preprocessor resolution = -1 100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [02:25<00:00, 5.83s/it] Total progress: 100%|██████████████████████████████████████████████████████████████████| 25/25 [02:24<00:00, 5.78s/it] 100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [01:44<00:00, 4.19s/it] Total progress: 100%|██████████████████████████████████████████████████████████████████| 25/25 [01:44<00:00, 4.19s/it] Total progress: 100%|██████████████████████████████████████████████████████████████████| 25/25 [01:44<00:00, 4.11s/it]

lllyasviel commented 1 year ago

I cannot reproduce this problem but i will add help wanted and see if others have the problem

level999999 commented 1 year ago

同様の症状に悩んでいます。 Hires使用時に、controlnetのtileを使用した際に、以前の10倍近く生成時間がかかるようになりました。

controlnet未使用でHiresだけだと生成が長時間になることはありません。

justinwiggins commented 1 year ago

I have the exact same problem, my renders used to take 22 seconds on Euler A. As soon as I add controlnet, it takes 12-16 min. i thought it might have been Tiled VAE. But disabled that too, no luck.

0xBADDCAFE commented 10 months ago

Same problem occurred; low VRAM (< 8GB) seems to be causing this problem. Rendering with ControlNet consumes VRAM near (or over) 8GB and it doesn't release.

Using

to avoid this problem.