search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.68k stars 422 forks source link

[Extension]: ControlNet broken for inpainting "only masked" on fresh install #2568

Closed TimothyKrell closed 8 months ago

TimothyKrell commented 11 months ago

Issue Description

After a fresh install, I can't use ControlNet with inpainting with "only masked" setting. It works fine with img2img and inpainting "whole picture", though.

I used openpose and inpaint masked.

Version Platform Description

Python 3.10.6 on Windows Version: app=sd.next updated=2023-11-27 hash=8136723f url=https://github.com/vladmandic/automatic/tree/master Platform: arch=AMD64 cpu=AMD64 Family 25 Model 33 Stepping 0, AuthenticAMD system=Windows release=Windows-10-10.0.19045-SP0 python=3.10.6 Device: device=NVIDIA GeForce RTX 3090 n=1 arch=sm_90 cap=(8, 6) cuda=12.1 cudnn=8801 driver=536.23

Relevant log output

08:04:12-812150 ERROR    Running script process: extensions-builtin\sd-webui-controlnet\scripts\controlnet.py: IndexError
╭─────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────╮
│ C:\Users\timot\Code\sdnext3\modules\scripts.py:519 in process                                                            │
│                                                                                                                          │
│   518 │   │   │   │   args = p.per_script_args.get(script.title(), p.script_args[script.args_f                           │
│ ❱ 519 │   │   │   │   script.process(p, *args, **kwargs)                                                                 │
│   520 │   │   │   except Exception as e:                                                                                 │
│                                                                                                                          │
│ C:\Users\timot\Code\sdnext3\extensions-builtin\sd-webui-controlnet\scripts\controlnet.py:993 in process                  │
│                                                                                                                          │
│    992 │   │   if not Script.process_has_sdxl_refiner(p):                                                                │
│ ❱  993 │   │   │   self.controlnet_hack(p)                                                                               │
│    994 │   │   return                                                                                                    │
│                                                                                                                          │
│ C:\Users\timot\Code\sdnext3\extensions-builtin\sd-webui-controlnet\scripts\controlnet.py:982 in controlnet_hack          │
│                                                                                                                          │
│    981 │   │   t = time.time()                                                                                           │
│ ❱  982 │   │   self.controlnet_main_entry(p)                                                                             │
│    983 │   │   if len(self.enabled_units) > 0:                                                                           │
│                                                                                                                          │
│ C:\Users\timot\Code\sdnext3\extensions-builtin\sd-webui-controlnet\scripts\controlnet.py:747 in controlnet_main_entry    │
│                                                                                                                          │
│    746 │   │   │   │                                                                                                     │
│ ❱  747 │   │   │   │   input_image = [np.asarray(x)[:, :, 0] for x in input_image]                                       │
│    748 │   │   │   │   input_image = np.stack(input_image, axis=2)                                                       │
│                                                                                                                          │
│ C:\Users\timot\Code\sdnext3\extensions-builtin\sd-webui-controlnet\scripts\controlnet.py:747 in <listcomp>               │
│                                                                                                                          │
│    746 │   │   │   │                                                                                                     │
│ ❱  747 │   │   │   │   input_image = [np.asarray(x)[:, :, 0] for x in input_image]                                       │
│    748 │   │   │   │   input_image = np.stack(input_image, axis=2)                                                       │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed

Backend

Original

Branch

Master

Model

SD 1.5

Acknowledgements

TimothyKrell commented 11 months ago

I'm not sure if I classified this issue right. I probably should have written it as an "extension" issue, but since controlnet is included by default, I wasn't sure.

vladmandic commented 11 months ago

if issue is not reproducible without controlnet, then its an extension issue. but ok, leave it and i'll reclassify.

Turokosh commented 11 months ago

Same problem, but using Ultimate Upscaler and controlnet in img2img. Cannot be reproduced in A1111. https://github.com/Mikubill/sd-webui-controlnet/issues/2285

vladmandic commented 11 months ago

Same problem, but using Ultimate Upscaler and controlnet in img2img. Cannot be reproduced in A1111. Mikubill/sd-webui-controlnet#2285

I need exact and detailed workflow as I cannot guess what exactly you're doing nor I can click around randomly until I get to something that may look like your workflow.

Turokosh commented 11 months ago

Same problem, but using Ultimate Upscaler and controlnet in img2img. Cannot be reproduced in A1111. Mikubill/sd-webui-controlnet#2285

I need exact and detailed workflow as I cannot guess what exactly you're doing nor I can click around randomly until I get to something that may look like your workflow.

Sorry, I will provide the workflow where I have an error.

Version Platform Description Python 3.10.12 on Linux Version: app=sd.next updated=2023-11-30 hash=c957a2fb url=https://github.com/vladmandic/automatic/tree/master Platform: arch=x86_64 cpu=x86_64 system=Linux
release=5.15.0-89-generic python=3.10.12, Device: device=AMD Radeon RX 6900 XT n=1
hip=5.5.30201-c1741e9b

Workflow

  1. Generate txt2img with default settings
  2. Import a generated image to img2img (default setting)
  3. Set Script/Ultimate SD upscale. Target size type/Scale from image size. Scale/2x. Redraw options: ESRGAN 4x NMKD Siax, Type/Linear, Tile width/512, Tile height/0, Mask blur/8, Padding/32, Seams fix: None, Save options: Upscaled (yes)
  4. Set ControlNet v1.1.419, Enable (yes), Pixel Perfect (yes), Control Type: Tile/blur, Preprocessor/tile_resample, Model/control_v11f1e_sd15_tile [a371b31b], Control Weight/1, Starting Control Step/0, Ending Control Step/1, Down Sampling Rate/1, Control Mode/Balanced 4.1. (I don’t always use it, but there’s an error there too) Upload independent control image/Allow Preview/load image/Run preprocessor
  5. Generate
  6. ERROR
    ERROR 

20:56:30-027954 ERROR    Running script process:                                
                         extensions-builtin/sd-webui-controlnet/scripts/controln
                         et.py: IndexError                                      
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/Turokosh/Table/automatic/modules/scripts.py:519 in process                 │
│                                                                              │
│   518 │   │   │   │   args = p.per_script_args.get(script.title(), p.script_ │
│ ❱ 519 │   │   │   │   script.process(p, *args, **kwargs)                     │
│   520 │   │   │   except Exception as e:                                     │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:993 in process                                                   │
│                                                                              │
│    992 │   │   if not Script.process_has_sdxl_refiner(p):                    │
│ ❱  993 │   │   │   self.controlnet_hack(p)                                   │
│    994 │   │   return                                                        │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:982 in controlnet_hack                                           │
│                                                                              │
│    981 │   │   t = time.time()                                               │
│ ❱  982 │   │   self.controlnet_main_entry(p)                                 │
│    983 │   │   if len(self.enabled_units) > 0:                               │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:747 in controlnet_main_entry                                     │
│                                                                              │
│    746 │   │   │   │                                                         │
│ ❱  747 │   │   │   │   input_image = [np.asarray(x)[:, :, 0] for x in input_ │
│    748 │   │   │   │   input_image = np.stack(input_image, axis=2)           │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:747 in <listcomp>                                                │
│                                                                              │
│    746 │   │   │   │                                                         │
│ ❱  747 │   │   │   │   input_image = [np.asarray(x)[:, :, 0] for x in input_ │
│    748 │   │   │   │   input_image = np.stack(input_image, axis=2)           │
╰──────────────────────────────────────────────────────────────────────────────╯
IndexError: too many indices for array: array is 2-dimensional, but 3 were 
indexed
20:56:30-194971 DEBUG    Sampler: sampler="UniPC" config={}                     
Progress 5.0it/s ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:03
20:56:33-902313 INFO     Processed: images=1 time=3.71 its=5.39 memory={'ram':  
                         {'used': 7.48, 'total': 62.54}, 'gpu': {'used': 2.6,   
                         'total': 15.98}, 'retries': 0, 'oom': 0}               
20:56:33-922968 ERROR    Running script process:                                
                         extensions-builtin/sd-webui-controlnet/scripts/controln
                         et.py: IndexError                                      
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/Turokosh/Table/automatic/modules/scripts.py:519 in process                 │
│                                                                              │
│   518 │   │   │   │   args = p.per_script_args.get(script.title(), p.script_ │
│ ❱ 519 │   │   │   │   script.process(p, *args, **kwargs)                     │
│   520 │   │   │   except Exception as e:                                     │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:993 in process                                                   │
│                                                                              │
│    992 │   │   if not Script.process_has_sdxl_refiner(p):                    │
│ ❱  993 │   │   │   self.controlnet_hack(p)                                   │
│    994 │   │   return                                                        │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:982 in controlnet_hack                                           │
│                                                                              │
│    981 │   │   t = time.time()                                               │
│ ❱  982 │   │   self.controlnet_main_entry(p)                                 │
│    983 │   │   if len(self.enabled_units) > 0:                               │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:747 in controlnet_main_entry                                     │
│                                                                              │
│    746 │   │   │   │                                                         │
│ ❱  747 │   │   │   │   input_image = [np.asarray(x)[:, :, 0] for x in input_ │
│    748 │   │   │   │   input_image = np.stack(input_image, axis=2)           │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:747 in <listcomp>                                                │
│                                                                              │
│    746 │   │   │   │                                                         │
│ ❱  747 │   │   │   │   input_image = [np.asarray(x)[:, :, 0] for x in input_ │
│    748 │   │   │   │   input_image = np.stack(input_image, axis=2)           │
╰──────────────────────────────────────────────────────────────────────────────╯
IndexError: too many indices for array: array is 2-dimensional, but 3 were 
indexed
20:56:34-074205 DEBUG    Sampler: sampler="UniPC" config={}                     
Progress 5.33it/s ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:03
20:56:37-460316 INFO     Processed: images=1 time=3.39 its=5.91 memory={'ram':  
                         {'used': 7.48, 'total': 62.54}, 'gpu': {'used': 3.27,  
                         'total': 15.98}, 'retries': 0, 'oom': 0}               
20:56:37-481006 ERROR    Running script process:                                
                         extensions-builtin/sd-webui-controlnet/scripts/controln
                         et.py: IndexError                                      
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/Turokosh/Table/automatic/modules/scripts.py:519 in process                 │
│                                                                              │
│   518 │   │   │   │   args = p.per_script_args.get(script.title(), p.script_ │
│ ❱ 519 │   │   │   │   script.process(p, *args, **kwargs)                     │
│   520 │   │   │   except Exception as e:                                     │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:993 in process                                                   │
│                                                                              │
│    992 │   │   if not Script.process_has_sdxl_refiner(p):                    │
│ ❱  993 │   │   │   self.controlnet_hack(p)                                   │
│    994 │   │   return                                                        │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:982 in controlnet_hack                                           │
│                                                                              │
│    981 │   │   t = time.time()                                               │
│ ❱  982 │   │   self.controlnet_main_entry(p)                                 │
│    983 │   │   if len(self.enabled_units) > 0:                               │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:747 in controlnet_main_entry                                     │
│                                                                              │
│    746 │   │   │   │                                                         │
│ ❱  747 │   │   │   │   input_image = [np.asarray(x)[:, :, 0] for x in input_ │
│    748 │   │   │   │   input_image = np.stack(input_image, axis=2)           │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:747 in <listcomp>                                                │
│                                                                              │
│    746 │   │   │   │                                                         │
│ ❱  747 │   │   │   │   input_image = [np.asarray(x)[:, :, 0] for x in input_ │
│    748 │   │   │   │   input_image = np.stack(input_image, axis=2)           │
╰──────────────────────────────────────────────────────────────────────────────╯
IndexError: too many indices for array: array is 2-dimensional, but 3 were 
indexed
20:56:37-636363 DEBUG    Sampler: sampler="UniPC" config={}                     
Progress 5.33it/s ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:03
20:56:41-023913 INFO     Processed: images=1 time=3.39 its=5.90 memory={'ram':  
                         {'used': 7.48, 'total': 62.54}, 'gpu': {'used': 3.27,  
                         'total': 15.98}, 'retries': 0, 'oom': 0}               
20:56:41-044789 ERROR    Running script process:                                
                         extensions-builtin/sd-webui-controlnet/scripts/controln
                         et.py: IndexError                                      
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/Turokosh/Table/automatic/modules/scripts.py:519 in process                 │
│                                                                              │
│   518 │   │   │   │   args = p.per_script_args.get(script.title(), p.script_ │
│ ❱ 519 │   │   │   │   script.process(p, *args, **kwargs)                     │
│   520 │   │   │   except Exception as e:                                     │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:993 in process                                                   │
│                                                                              │
│    992 │   │   if not Script.process_has_sdxl_refiner(p):                    │
│ ❱  993 │   │   │   self.controlnet_hack(p)                                   │
│    994 │   │   return                                                        │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:982 in controlnet_hack                                           │
│                                                                              │
│    981 │   │   t = time.time()                                               │
│ ❱  982 │   │   self.controlnet_main_entry(p)                                 │
│    983 │   │   if len(self.enabled_units) > 0:                               │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:747 in controlnet_main_entry                                     │
│                                                                              │
│    746 │   │   │   │                                                         │
│ ❱  747 │   │   │   │   input_image = [np.asarray(x)[:, :, 0] for x in input_ │
│    748 │   │   │   │   input_image = np.stack(input_image, axis=2)           │
│                                                                              │
│ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │
│ ntrolnet.py:747 in <listcomp>                                                │
│                                                                              │
│    746 │   │   │   │                                                         │
│ ❱  747 │   │   │   │   input_image = [np.asarray(x)[:, :, 0] for x in input_ │
│    748 │   │   │   │   input_image = np.stack(input_image, axis=2)           │
╰──────────────────────────────────────────────────────────────────────────────╯
IndexError: too many indices for array: array is 2-dimensional, but 3 were 
indexed
20:56:41-198634 DEBUG    Sampler: sampler="UniPC" config={}                     
Progress 5.33it/s ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:03
20:56:44-588044 INFO     Processed: images=1 time=3.39 its=5.90 memory={'ram':  
                         {'used': 7.48, 'total': 62.54}, 'gpu': {'used': 3.27,  
                         'total': 15.98}, 'retries': 0, 'oom': 0}               
20:56:44-589412 DEBUG    Saving:                                                
                         image="outputs/image/00008-v1-5-pruned-emaonly-cat.jpg"
                         type=JPEG size=1024x1024

All log output (debug)

```python 20:55:30-214557 DEBUG Logger: file=/home/Turokosh/Table/automatic/sdnext.log level=10 size=0 mode=create 20:55:30-216348 INFO Starting SD.Next 20:55:30-216977 INFO Python 3.10.12 on Linux 20:55:30-221302 INFO Version: app=sd.next updated=2023-11-30 hash=c957a2fb url=https://github.com/vladmandic/automatic/tree/master 20:55:30-587632 INFO Platform: arch=x86_64 cpu=x86_64 system=Linux release=5.15.0-89-generic python=3.10.12 20:55:30-591673 DEBUG Setting environment tuning 20:55:30-594329 DEBUG Cache folder: /home/Turokosh/.cache/huggingface/hub 20:55:30-596949 DEBUG Torch overrides: cuda=False rocm=False ipex=False diml=False openvino=False 20:55:30-599555 DEBUG Torch allowed: cuda=True rocm=True ipex=True diml=True openvino=True 20:55:30-602211 INFO AMD ROCm toolkit detected 20:55:30-700600 DEBUG ROCm agents detected: ['gfx1030'] 20:55:30-701205 DEBUG ROCm agent used by default: idx=0 gpu=gfx1030 arch=navi2x 20:55:30-702078 DEBUG ROCm hipconfig failed: local variable 'rocm_ver' referenced before assignment 20:55:30-710584 DEBUG Repository update time: Thu Nov 30 10:18:05 2023 20:55:30-713085 INFO Startup: standard 20:55:30-714768 INFO Verifying requirements 20:55:30-727877 INFO Verifying packages 20:55:30-728890 INFO Verifying submodules 20:55:31-267633 DEBUG Submodule: extensions-builtin/sd-extension-chainner / main 20:55:31-284667 DEBUG Submodule: extensions-builtin/sd-extension-system-info / main 20:55:31-301108 DEBUG Submodule: extensions-builtin/sd-webui-agent-scheduler / main 20:55:31-317514 DEBUG Submodule: extensions-builtin/sd-webui-controlnet / main 20:55:31-337218 DEBUG Submodule: extensions-builtin/stable-diffusion-webui-images-browse r / main 20:55:31-350407 DEBUG Submodule: extensions-builtin/stable-diffusion-webui-rembg / master 20:55:31-366662 DEBUG Submodule: modules/k-diffusion / master 20:55:31-382314 DEBUG Submodule: modules/lora / main 20:55:31-399032 DEBUG Submodule: wiki / master 20:55:31-411229 DEBUG Register paths 20:55:31-452730 DEBUG Installed packages: 215 20:55:31-453214 DEBUG Extensions all: ['stable-diffusion-webui-rembg', 'sd-extension-chainner', 'sd-extension-system-info', 'Lora', 'stable-diffusion-webui-images-browser', 'sd-webui-agent-scheduler', 'sd-webui-controlnet'] 20:55:31-453787 DEBUG Running extension installer: /home/Turokosh/Table/automatic/extensions-builtin/stable-di ffusion-webui-rembg/install.py 20:55:31-562447 DEBUG Running extension installer: /home/Turokosh/Table/automatic/extensions-builtin/sd-extens ion-system-info/install.py 20:55:31-672093 DEBUG Running extension installer: /home/Turokosh/Table/automatic/extensions-builtin/stable-di ffusion-webui-images-browser/install.py 20:55:31-760271 DEBUG Running extension installer: /home/Turokosh/Table/automatic/extensions-builtin/sd-webui- agent-scheduler/install.py 20:55:31-847970 DEBUG Running extension installer: /home/Turokosh/Table/automatic/extensions-builtin/sd-webui- controlnet/install.py 20:55:31-936310 DEBUG Extensions all: ['ultimate-upscale-for-automatic1111'] 20:55:31-956294 INFO Extensions enabled: ['stable-diffusion-webui-rembg', 'sd-extension-chainner', 'sd-extension-system-info', 'Lora', 'stable-diffusion-webui-images-browser', 'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'ultimate-upscale-for-automatic1111'] 20:55:31-957005 INFO Verifying requirements 20:55:31-964023 DEBUG Setup complete without errors: 1701370532 20:55:31-965498 INFO Extension preload: {'extensions-builtin': 0.0, 'extensions': 0.0} 20:55:31-966190 DEBUG Starting module: 20:55:31-966813 INFO Command line args: ['--debug'] debug=True 20:55:33-667969 INFO Load packages: torch=2.1.1+rocm5.5 diffusers=0.23.1 gradio=3.43.2 20:55:33-901423 DEBUG Read: file="config.json" json=16 bytes=688 20:55:33-903364 INFO Engine: backend=Backend.ORIGINAL compute=rocm mode=no_grad device=cuda cross-optimization="Sub-quadratic" 20:55:33-904345 INFO Device: device=AMD Radeon RX 6900 XT n=1 hip=5.5.30201-c1741e9b 20:55:34-939133 DEBUG Entering start sequence 20:55:34-940051 DEBUG Initializing 20:55:34-940918 INFO Available VAEs: path="models/VAE" items=1 20:55:34-941454 INFO Disabling uncompatible extensions: backend=Backend.ORIGINAL [] 20:55:34-942126 DEBUG Read: file="cache.json" json=1 bytes=351 20:55:34-942667 DEBUG Read: file="metadata.json" json=1 bytes=163 20:55:34-943149 INFO Available models: path="models/Stable-diffusion" items=1 time=0.00 20:55:35-021454 DEBUG Load extensions 20:55:35-480045 INFO Extension: script='extensions-builtin/sd-webui-agent-scheduler/scr ipts/task_scheduler.py' Using sqlite file: extensions-builtin/sd-webui-agent-scheduler/task_schedu ler.sqlite3 20:55:35-617856 INFO Extension: script='extensions-builtin/sd-webui-controlnet/scripts/ controlnet.py' Warning: ControlNet failed to load SGM - will use LDM instead. 20:55:35-618547 INFO Extension: script='extensions-builtin/sd-webui-controlnet/scripts/ controlnet.py' ControlNet preprocessor location: /home/Turokosh/Table/automatic/extensions-builtin/sd-webui- controlnet/annotator/downloads 20:55:35-622025 INFO Extension: script='extensions-builtin/sd-webui-controlnet/scripts/ hook.py' Warning: ControlNet failed to load SGM - will use LDM instead. 20:55:36-015428 INFO Extensions time: 0.99 { Lora=0.24 sd-webui-agent-scheduler=0.17 sd-webui-controlnet=0.14 stable-diffusion-webui-rembg=0.35 } 20:55:36-060150 DEBUG Read: file="html/upscalers.json" json=4 bytes=2640 20:55:36-060859 DEBUG Read: file="extensions-builtin/sd-extension-chainner/models.j son" json=24 bytes=2693 20:55:36-061630 DEBUG chaiNNer models: path="models/chaiNNer" defined=24 discovered=0 downloaded=1 20:55:36-062446 DEBUG Load upscalers: total=50 downloaded=3 user=0 ['None', 'Lanczos', 'Nearest', 'ChaiNNer', 'RealESRGAN', 'LDSR', 'SwinIR', 'SCUNet', 'SD', 'ESRGAN'] 20:55:36-065179 DEBUG Load styles: folder="models/styles" items=288 20:55:36-066221 DEBUG Creating UI 20:55:36-253822 INFO Load UI theme: name="black-teal" style=Auto base=sdnext.css 20:55:36-288330 DEBUG Extra networks: page='model' items=1 subfolders=1 tab=txt2img folders=['models/Stable-diffusion', 'models/Diffusers', 'models/Reference', '/home/Turokosh/Table/automatic/models/Stable-diffusion'] list=0.00 desc=0.00 info=0.00 20:55:36-293200 DEBUG Extra networks: page='style' items=288 subfolders=2 tab=txt2img folders=['models/styles', 'html'] list=0.00 desc=0.00 info=0.00 20:55:36-294090 DEBUG Extra networks: page='embedding' items=0 subfolders=1 tab=txt2img folders=['models/embeddings'] list=0.00 desc=0.00 info=0.00 20:55:36-294936 DEBUG Extra networks: page='hypernetwork' items=0 subfolders=1 tab=txt2img folders=['models/hypernetworks'] list=0.00 desc=0.00 info=0.00 20:55:36-295771 DEBUG Extra networks: page='vae' items=1 subfolders=1 tab=txt2img folders=['models/VAE'] list=0.00 desc=0.00 info=0.00 20:55:36-296555 DEBUG Extra networks: page='lora' items=0 subfolders=1 tab=txt2img folders=['models/Lora', 'models/LyCORIS'] list=0.00 desc=0.00 info=0.00 20:55:36-433557 DEBUG Read: file="ui-config.json" json=0 bytes=2 20:55:36-496151 DEBUG Themes: builtin=6 default=5 external=55 20:55:36-793158 DEBUG Script: 0.26 ui_tabs /home/Turokosh/Table/automatic/extensions-builtin/stable-di ffusion-webui-images-browser/scripts/image_browser.py 20:55:36-815080 DEBUG Extension list: processed=8 installed=8 enabled=8 disabled=0 visible=8 hidden=0 20:55:37-011223 INFO Local URL: http://127.0.0.1:7860/ 20:55:37-011993 DEBUG Gradio functions: registered=2081 20:55:37-012493 INFO Initializing middleware 20:55:37-014720 DEBUG Creating API 20:55:37-115110 INFO [AgentScheduler] Task queue is empty 20:55:37-115732 INFO [AgentScheduler] Registering APIs 20:55:37-180929 DEBUG Scripts setup: ['ControlNet:0.037'] 20:55:37-182826 DEBUG Model metadata: file="metadata.json" no changes 20:55:37-183275 DEBUG Model auto load disabled 20:55:37-183832 DEBUG Save: file="config.json" json=16 bytes=688 20:55:37-184314 INFO Startup time: 5.21 { torch=1.30 gradio=0.39 libraries=1.27 extensions=0.99 face-restore=0.08 ui-extra-networks=0.23 ui-img2img=0.06 ui-settings=0.10 ui-extensions=0.30 launch=0.15 app-started=0.12 } 20:55:40-540308 INFO MOTD: N/A 20:55:42-185653 DEBUG Themes: builtin=6 default=5 external=55 20:55:42-330330 INFO Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 20:55:46-386445 DEBUG txt2img: id_task=task(3u0ofw53x4y66w7)|prompt=cat|negative_promp t=|prompt_styles=[]|steps=20|sampler_index=None|latent_ index=None|full_quality=True|restore_faces=False|tiling =False|n_iter=1|batch_size=1|cfg_scale=6|clip_skip=1|se ed=-1.0|subseed=-1.0|subseed_strength=0|seed_resize_fro m_h=0|seed_resize_from_w=0||height=512|width=512|enable _hr=False|denoising_strength=0.5|hr_scale=2|hr_upscaler =None|hr_force=False|hr_second_pass_steps=20|hr_resize_ x=0|hr_resize_y=0|image_cfg_scale=6|diffusers_guidance_ rescale=0.7|refiner_steps=5|refiner_start=0.8|refiner_p rompt=|refiner_negative=|override_settings_texts=[] 20:55:46-388245 INFO Select: model="v1-5-pruned-emaonly [cc6cb27103]" 20:55:46-388973 DEBUG Load model weights: existing=False target=/home/Turokosh/Table/automatic/models/Stable-diffusi on/v1-5-pruned-emaonly.ckpt info=None Loading model: /home/Turokosh/Table/automatic/models/Stable-diffusion/v1-5-pruned-… 20:55:47-471116 DEBUG Load model: name=/home/Turokosh/Table/automatic/models/Stable-diffusion /v1-5-pruned-emaonly.ckpt dict=True 20:55:47-603335 DEBUG Desired Torch parameters: dtype=FP16 no-half=False no-half-vae=False upscast=False 20:55:47-604102 INFO Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16 context=no_grad fp16=True bf16=False 20:55:47-604971 DEBUG Model dict loaded: {'ram': {'used': 5.51, 'total': 62.54}, 'gpu': {'used': 0.08, 'total': 15.98}, 'retries': 0, 'oom': 0} 20:55:47-612261 DEBUG Model config loaded: {'ram': {'used': 5.51, 'total': 62.54}, 'gpu': {'used': 0.08, 'total': 15.98}, 'retries': 0, 'oom': 0} 20:55:52-086825 INFO LDM: LatentDiffusion: Running in eps-prediction mode 20:55:52-087600 INFO LDM: DiffusionWrapper has 859.52 M params. 20:55:52-088071 INFO LDM: LatentDiffusion: Running in eps-prediction mode 20:55:52-088484 INFO LDM: DiffusionWrapper has 859.52 M params. 20:55:52-088935 DEBUG Model created from config: /home/Turokosh/Table/automatic/configs/v1-inference.yaml 20:55:52-089592 DEBUG Model weights loading: {'ram': {'used': 10.31, 'total': 62.54}, 'gpu': {'used': 0.08, 'total': 15.98}, 'retries': 0, 'oom': 0} Loading model: models/VAE/vae-ft-mse-840000-ema-pruned.safetensors ━━━ 0.0… -:-… MB 20:55:52-661186 DEBUG Model weights loaded: {'ram': {'used': 9.32, 'total': 62.54}, 'gpu': {'used': 0.08, 'total': 15.98}, 'retries': 0, 'oom': 0} 20:55:52-804678 DEBUG Model weights moved: {'ram': {'used': 8.93, 'total': 62.54}, 'gpu': {'used': 2.11, 'total': 15.98}, 'retries': 0, 'oom': 0} 20:55:52-809217 INFO Cross-attention: optimization=Sub-quadratic options=[] 20:55:52-871843 INFO Load embeddings: loaded=0 skipped=0 time=0.06 20:55:52-874144 INFO Model loaded in 6.48 { load=1.08 config=0.14 create=4.48 apply=0.30 vae=0.27 move=0.14 embeddings=0.06 } 20:55:53-046641 DEBUG gc: collected=19659 device=cuda {'ram': {'used': 8.28, 'total': 62.54}, 'gpu': {'used': 2.11, 'total': 15.98}, 'retries': 0, 'oom': 0} 20:55:53-047672 INFO Model load finished: {'ram': {'used': 8.28, 'total': 62.54}, 'gpu': {'used': 2.11, 'total': 15.98}, 'retries': 0, 'oom': 0} cached=0 20:55:53-253687 DEBUG Sampler: sampler="UniPC" config={} Progress 1.83it/s ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5% -:--:-- 0:00:0020:55:53-956507 DEBUG Load VAE decode approximate: model="models/VAE-approx/model.pt" Progress 6.98it/s ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:02 20:55:56-295212 DEBUG Saving: image="outputs/text/00006-v1-5-pruned-emaonly-cat.jpg" type=JPEG size=512x512 20:55:56-300683 INFO Processed: images=1 time=3.22 its=6.22 memory={'ram': {'used': 6.19, 'total': 62.54}, 'gpu': {'used': 2.49, 'total': 15.98}, 'retries': 0, 'oom': 0} 20:56:00-210422 DEBUG Server: alive=True jobs=1 requests=48 uptime=26 memory=6.19/62.54 backend=Backend.ORIGINAL state=job="txt2img" 0/-1 20:56:27-541389 DEBUG img2img: id_task=task(hlywdnh3z5gvciw)|mode=0|prompt=cat|negativ e_prompt=|prompt_styles=[]|init_img=|sketch=None|init_img_with_mask=None|inp aint_color_sketch=None|inpaint_color_sketch_orig=None|i nit_img_inpaint=None|init_mask_inpaint=None|steps=20|sa mpler_index=None|latent_index=None|mask_blur=4|mask_alp ha=0|inpainting_fill=1|full_quality=True|restore_faces= False|tiling=False|n_iter=1|batch_size=1|cfg_scale=6|im age_cfg_scale=1.5|clip_skip=1|denoising_strength=0.75|s eed=-1.0|subseed-1.0|subseed_strength=0|seed_resize_fro m_h=0|seed_resize_from_w=0|selected_scale_tab=0|height= 512|width=512|scale_by=1|resize_mode=0|inpaint_full_res =0|inpaint_full_res_padding=32|inpainting_mask_invert=0 |img2img_batch_files=None|img2img_batch_input_dir=|img2 img_batch_output_dir=|img2img_batch_inpaint_mask_dir=|o verride_settings_texts=[] Canva size: 1024x1024 Image size: 512x512 Scale factor: 2 Upscaling iteration 1 with scale factor 2 20:56:27-568068 INFO Upscaler loaded: type=ESRGAN model=models/ESRGAN/ESRGAN-NMKD-Siax-4x.pth Upscaling ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:00 Tile size: 512x512 Tiles amount: 4 Grid: 2x2 Redraw enabled: True Seams fix mode: NONE 20:56:30-027954 ERROR Running script process: extensions-builtin/sd-webui-controlnet/scripts/controln et.py: IndexError ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /home/Turokosh/Table/automatic/modules/scripts.py:519 in process │ │ │ │ 518 │ │ │ │ args = p.per_script_args.get(script.title(), p.script_ │ │ ❱ 519 │ │ │ │ script.process(p, *args, **kwargs) │ │ 520 │ │ │ except Exception as e: │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:993 in process │ │ │ │ 992 │ │ if not Script.process_has_sdxl_refiner(p): │ │ ❱ 993 │ │ │ self.controlnet_hack(p) │ │ 994 │ │ return │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:982 in controlnet_hack │ │ │ │ 981 │ │ t = time.time() │ │ ❱ 982 │ │ self.controlnet_main_entry(p) │ │ 983 │ │ if len(self.enabled_units) > 0: │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:747 in controlnet_main_entry │ │ │ │ 746 │ │ │ │ │ │ ❱ 747 │ │ │ │ input_image = [np.asarray(x)[:, :, 0] for x in input_ │ │ 748 │ │ │ │ input_image = np.stack(input_image, axis=2) │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:747 in │ │ │ │ 746 │ │ │ │ │ │ ❱ 747 │ │ │ │ input_image = [np.asarray(x)[:, :, 0] for x in input_ │ │ 748 │ │ │ │ input_image = np.stack(input_image, axis=2) │ ╰──────────────────────────────────────────────────────────────────────────────╯ IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed 20:56:30-194971 DEBUG Sampler: sampler="UniPC" config={} Progress 5.0it/s ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:03 20:56:33-902313 INFO Processed: images=1 time=3.71 its=5.39 memory={'ram': {'used': 7.48, 'total': 62.54}, 'gpu': {'used': 2.6, 'total': 15.98}, 'retries': 0, 'oom': 0} 20:56:33-922968 ERROR Running script process: extensions-builtin/sd-webui-controlnet/scripts/controln et.py: IndexError ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /home/Turokosh/Table/automatic/modules/scripts.py:519 in process │ │ │ │ 518 │ │ │ │ args = p.per_script_args.get(script.title(), p.script_ │ │ ❱ 519 │ │ │ │ script.process(p, *args, **kwargs) │ │ 520 │ │ │ except Exception as e: │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:993 in process │ │ │ │ 992 │ │ if not Script.process_has_sdxl_refiner(p): │ │ ❱ 993 │ │ │ self.controlnet_hack(p) │ │ 994 │ │ return │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:982 in controlnet_hack │ │ │ │ 981 │ │ t = time.time() │ │ ❱ 982 │ │ self.controlnet_main_entry(p) │ │ 983 │ │ if len(self.enabled_units) > 0: │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:747 in controlnet_main_entry │ │ │ │ 746 │ │ │ │ │ │ ❱ 747 │ │ │ │ input_image = [np.asarray(x)[:, :, 0] for x in input_ │ │ 748 │ │ │ │ input_image = np.stack(input_image, axis=2) │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:747 in │ │ │ │ 746 │ │ │ │ │ │ ❱ 747 │ │ │ │ input_image = [np.asarray(x)[:, :, 0] for x in input_ │ │ 748 │ │ │ │ input_image = np.stack(input_image, axis=2) │ ╰──────────────────────────────────────────────────────────────────────────────╯ IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed 20:56:34-074205 DEBUG Sampler: sampler="UniPC" config={} Progress 5.33it/s ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:03 20:56:37-460316 INFO Processed: images=1 time=3.39 its=5.91 memory={'ram': {'used': 7.48, 'total': 62.54}, 'gpu': {'used': 3.27, 'total': 15.98}, 'retries': 0, 'oom': 0} 20:56:37-481006 ERROR Running script process: extensions-builtin/sd-webui-controlnet/scripts/controln et.py: IndexError ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /home/Turokosh/Table/automatic/modules/scripts.py:519 in process │ │ │ │ 518 │ │ │ │ args = p.per_script_args.get(script.title(), p.script_ │ │ ❱ 519 │ │ │ │ script.process(p, *args, **kwargs) │ │ 520 │ │ │ except Exception as e: │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:993 in process │ │ │ │ 992 │ │ if not Script.process_has_sdxl_refiner(p): │ │ ❱ 993 │ │ │ self.controlnet_hack(p) │ │ 994 │ │ return │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:982 in controlnet_hack │ │ │ │ 981 │ │ t = time.time() │ │ ❱ 982 │ │ self.controlnet_main_entry(p) │ │ 983 │ │ if len(self.enabled_units) > 0: │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:747 in controlnet_main_entry │ │ │ │ 746 │ │ │ │ │ │ ❱ 747 │ │ │ │ input_image = [np.asarray(x)[:, :, 0] for x in input_ │ │ 748 │ │ │ │ input_image = np.stack(input_image, axis=2) │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:747 in │ │ │ │ 746 │ │ │ │ │ │ ❱ 747 │ │ │ │ input_image = [np.asarray(x)[:, :, 0] for x in input_ │ │ 748 │ │ │ │ input_image = np.stack(input_image, axis=2) │ ╰──────────────────────────────────────────────────────────────────────────────╯ IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed 20:56:37-636363 DEBUG Sampler: sampler="UniPC" config={} Progress 5.33it/s ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:03 20:56:41-023913 INFO Processed: images=1 time=3.39 its=5.90 memory={'ram': {'used': 7.48, 'total': 62.54}, 'gpu': {'used': 3.27, 'total': 15.98}, 'retries': 0, 'oom': 0} 20:56:41-044789 ERROR Running script process: extensions-builtin/sd-webui-controlnet/scripts/controln et.py: IndexError ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /home/Turokosh/Table/automatic/modules/scripts.py:519 in process │ │ │ │ 518 │ │ │ │ args = p.per_script_args.get(script.title(), p.script_ │ │ ❱ 519 │ │ │ │ script.process(p, *args, **kwargs) │ │ 520 │ │ │ except Exception as e: │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:993 in process │ │ │ │ 992 │ │ if not Script.process_has_sdxl_refiner(p): │ │ ❱ 993 │ │ │ self.controlnet_hack(p) │ │ 994 │ │ return │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:982 in controlnet_hack │ │ │ │ 981 │ │ t = time.time() │ │ ❱ 982 │ │ self.controlnet_main_entry(p) │ │ 983 │ │ if len(self.enabled_units) > 0: │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:747 in controlnet_main_entry │ │ │ │ 746 │ │ │ │ │ │ ❱ 747 │ │ │ │ input_image = [np.asarray(x)[:, :, 0] for x in input_ │ │ 748 │ │ │ │ input_image = np.stack(input_image, axis=2) │ │ │ │ /home/Turokosh/Table/automatic/extensions-builtin/sd-webui-controlnet/scripts/co │ │ ntrolnet.py:747 in │ │ │ │ 746 │ │ │ │ │ │ ❱ 747 │ │ │ │ input_image = [np.asarray(x)[:, :, 0] for x in input_ │ │ 748 │ │ │ │ input_image = np.stack(input_image, axis=2) │ ╰──────────────────────────────────────────────────────────────────────────────╯ IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed 20:56:41-198634 DEBUG Sampler: sampler="UniPC" config={} Progress 5.33it/s ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:03 20:56:44-588044 INFO Processed: images=1 time=3.39 its=5.90 memory={'ram': {'used': 7.48, 'total': 62.54}, 'gpu': {'used': 3.27, 'total': 15.98}, 'retries': 0, 'oom': 0} 20:56:44-589412 DEBUG Saving: image="outputs/image/00008-v1-5-pruned-emaonly-cat.jpg" type=JPEG size=1024x1024 ```

vladmandic commented 11 months ago

this seems to be issue inside controlnet - please report there. i've added my notes to controlnet code here:

                logger.debug("A1111 inpaint mask START")
                input_image = [input_image[:, :, i] for i in range(input_image.shape[2])] # split rgb image into 3 bw images
                input_image = [Image.fromarray(x) for x in input_image] # convert to 3 pil images model=L

                mask = prepare_mask(a1111_mask_image, p)

                crop_region = masking.get_crop_region(np.array(mask), p.inpaint_full_res_padding)
                crop_region = masking.expand_crop_region(crop_region, p.width, p.height, mask.width, mask.height)

                input_image = [ # resize all 3 images
                    images.resize_image(resize_mode.int_value(), i, mask.width, mask.height) 
                    for i in input_image
                ]

                input_image = [x.crop(crop_region) for x in input_image] # crop all 3 images
                input_image = [ # resize all 3 images
                    images.resize_image(external_code.ResizeMode.OUTER_FIT.int_value(), x, p.width, p.height) 
                    for x in input_image
                ]

                input_image = [np.asarray(x)[:, :, 0] for x in input_image] # but this AGAIN tries to read RGB channels - input_image here is 2D!
                input_image = np.stack(input_image, axis=2)
                logger.debug("A1111 inpaint mask END")
abhinavzspace commented 9 months ago

Is there a workaround for now?

vladmandic commented 8 months ago

closing issue here as it was closed upstream in controlnet repo.