[Extension]: Tiled Diffusion with ControlNet tile not working

GodEmperor785 commented 1 year ago

Issue Description

After the recent big update (the one at Update for 2023-10-17) Tiled Diffusion + ControlNet tile stopped working (tested with both extensions at newest available versions). Originally there were many issues in Tiled Diffusion related to recent changes in the extension but after fixes by extension team they were resolved (see https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111/issues/318) and Tiled diffusion seems to work with SD.Next again. But there is one more problem which is specific to SD.Next - in Tiled Diffusion there is support for ControlNet, but on SD.Next it seems broken. Issue doesn't happen on original A1111 webui (fresh install of newest master) with same versions of both extensions. If it is important: SD.Next with descried problems was also a fresh install (made at 19.10.2023 but with newest fixes as of writing of this issue)

Here are steps to reproduce the problem:

In "From image" tab upload an image or send one which was generated in "From text"
Enable Tiled diffusion extension
For case A select an upscaler (other settings can be left at defaults)
For case B don't select any upscaler (leave it at None) or set "Scale factor"=1 (other settings can be left at defaults)
Enable ControlNet and select control type "tile/blur", it should automatically select proper preprocessor and model (at least it does so for me)
Try generating an image, depending on upscaler chosen in Tiled Diffusion it will throw exception A or B (errors at the end of issue)

I also noticed during testing that images of size 1152x1152 or 1728x1728 when "upscaling" with no upscaler with default Tiled diffusion settings actually works with ControlNet. It still fails for 512x512 or 768x768.

From what I can see in github blame part of code that throws exception in case A was not modified during recent problematic refactor of Tiled Diffusion.

Issue seems to be related specifically to recent SD.Next updates and not Tiled Diffusion updates, I installed newest Tiled Diffusion (61c8114bc258dda964588f2e27c0442d9af58a02) and ControlNet (1.1.411 - a43e574254d19a362082bbd412f24aeef1beed47) on SD.Next from before the big updates (updated: 2023-10-01, hash: c6226605) and failing scenarios (even same images) worked fine on this older version with newest extension versions. To test this I had to copy https://github.com/vladmandic/automatic/blob/master/modules/shared_state.py from current master due to missing imports in new Tiled Diffusion, but apart from that everything else in SD.Next was from before the big update.

I also tried with older Tiled Diffusion (from before the problematic changes - https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111/commit/f9f8073e64f4e682838f255215039ba7884553bf) and newest SD.Next + CotrolNet but same problems happened

I can run more tests if it would be needed.

So it seems that something in recent updates of SD.Next broke this. Could you check it? It would be great to have it working again, combination Tiled Diffusion + ControlNet is one of the best upscaling method I know.

Related logs/errors for case A and B (newest extensions + newest SD.Next):

Case A (with upscaler - exception was for basic ESRGANx4, upscale factor left at 2, changing upscale factor doesn't seem to change the error)

16:07:59-466134 ERROR    gradio call: RuntimeError
┌───────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────┐
│ H:\AI\sd_next\modules\call_queue.py:34 in f                                                                         │
│                                                                                                                     │
│   33 │   │   │   try:                                                                                               │
│ > 34 │   │   │   │   res = func(*args, **kwargs)                                                                    │
│   35 │   │   │   │   progress.record_results(id_task, res)                                                          │
│                                                                                                                     │
│ H:\AI\sd_next\modules\img2img.py:219 in img2img                                                                     │
│                                                                                                                     │
│   218 │   │   if processed is None:                                                                                 │
│ > 219 │   │   │   processed = processing.process_images(p)                                                          │
│   220 │   p.close()                                                                                                 │
│                                                                                                                     │
│ H:\AI\sd_next\modules\processing.py:683 in process_images                                                           │
│                                                                                                                     │
│    682 │   │   │   with context_hypertile_vae(p), context_hypertile_unet(p):                                        │
│ >  683 │   │   │   │   res = process_images_inner(p)                                                                │
│    684 │   finally:                                                                                                 │
│                                                                                                                     │
│ H:\AI\sd_next\extensions-builtin\sd-webui-controlnet\scripts\batch_hijack.py:42 in processing_process_images_hijack │
│                                                                                                                     │
│    41 │   │   │   # we are not in batch mode, fallback to original function                                         │
│ >  42 │   │   │   return getattr(processing, '__controlnet_original_process_images_inner')(p,                       │
│    43                                                                                                               │
│                                                                                                                     │
│ H:\AI\sd_next\modules\processing.py:774 in process_images_inner                                                     │
│                                                                                                                     │
│    773 │   │   with devices.autocast():                                                                             │
│ >  774 │   │   │   p.init(p.all_prompts, p.all_seeds, p.all_subseeds)                                               │
│    775 │   │   if shared.state.job_count == -1:                                                                     │
│                                                                                                                     │
│                                               ... 2 frames hidden ...                                               │
│                                                                                                                     │
│ H:\AI\sd_next\extensions\multidiffusion-upscaler-for-automatic1111\scripts\tilediffusion.py:447 in                  │
│ create_sampler_hijack                                                                                               │
│                                                                                                                     │
│   446 │   │   if self.controlnet_script:                                                                            │
│ > 447 │   │   │   delegate.init_controlnet(self.controlnet_script, control_tensor_cpu)                              │
│   448 │   │   if self.stablesr_script:                                                                              │
│                                                                                                                     │
│ H:\AI\sd_next\extensions\multidiffusion-upscaler-for-automatic1111\tile_utils\utils.py:249 in wrapper               │
│                                                                                                                     │
│   248 │   def wrapper(*args, **kwargs):                                                                             │
│ > 249 │   │   return fn(*args, **kwargs)                                                                            │
│   250 │   return wrapper                                                                                            │
│                                                                                                                     │
│ H:\AI\sd_next\extensions\multidiffusion-upscaler-for-automatic1111\tile_methods\abstractdiffusion.py:464 in         │
│ init_controlnet                                                                                                     │
│                                                                                                                     │
│   463 │   │                                                                                                         │
│ > 464 │   │   self.prepare_controlnet_tensors()                                                                     │
│   465                                                                                                               │
│                                                                                                                     │
│ H:\AI\sd_next\extensions\multidiffusion-upscaler-for-automatic1111\tile_utils\utils.py:249 in wrapper               │
│                                                                                                                     │
│   248 │   def wrapper(*args, **kwargs):                                                                             │
│ > 249 │   │   return fn(*args, **kwargs)                                                                            │
│   250 │   return wrapper                                                                                            │
│                                                                                                                     │
│ H:\AI\sd_next\extensions\multidiffusion-upscaler-for-automatic1111\tile_methods\abstractdiffusion.py:515 in         │
│ prepare_controlnet_tensors                                                                                          │
│                                                                                                                     │
│   514 │   │   │   │   │   single_batch_tensors.append(control_tile)                                                 │
│ > 515 │   │   │   │   control_tile = torch.cat(single_batch_tensors, dim=0)                                         │
│   516 │   │   │   │   if self.control_tensor_cpu:                                                                   │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 768 but got size 576 for tensor number 3
in the list.

Case B (no upscaler or "Scale factor"=1)

16:11:04-824335 ERROR    gradio call: RuntimeError
┌───────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────┐
│ H:\AI\sd_next\modules\call_queue.py:34 in f                                                                         │
│                                                                                                                     │
│   33 │   │   │   try:                                                                                               │
│ > 34 │   │   │   │   res = func(*args, **kwargs)                                                                    │
│   35 │   │   │   │   progress.record_results(id_task, res)                                                          │
│                                                                                                                     │
│ H:\AI\sd_next\modules\img2img.py:219 in img2img                                                                     │
│                                                                                                                     │
│   218 │   │   if processed is None:                                                                                 │
│ > 219 │   │   │   processed = processing.process_images(p)                                                          │
│   220 │   p.close()                                                                                                 │
│                                                                                                                     │
│ H:\AI\sd_next\modules\processing.py:683 in process_images                                                           │
│                                                                                                                     │
│    682 │   │   │   with context_hypertile_vae(p), context_hypertile_unet(p):                                        │
│ >  683 │   │   │   │   res = process_images_inner(p)                                                                │
│    684 │   finally:                                                                                                 │
│                                                                                                                     │
│ H:\AI\sd_next\extensions-builtin\sd-webui-controlnet\scripts\batch_hijack.py:42 in processing_process_images_hijack │
│                                                                                                                     │
│    41 │   │   │   # we are not in batch mode, fallback to original function                                         │
│ >  42 │   │   │   return getattr(processing, '__controlnet_original_process_images_inner')(p,                       │
│    43                                                                                                               │
│                                                                                                                     │
│ H:\AI\sd_next\modules\processing.py:818 in process_images_inner                                                     │
│                                                                                                                     │
│    817 │   │   │   │   with devices.without_autocast() if devices.unet_needs_upcast else device                     │
│ >  818 │   │   │   │   │   samples_ddim = p.sample(conditioning=c, unconditional_conditioning=u                     │
│    819 │   │   │   │   x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(                     │
│                                                                                                                     │
│                                              ... 26 frames hidden ...                                               │
│                                                                                                                     │
│ H:\AI\sd_next\venv\lib\site-packages\torch\nn\modules\module.py:1527 in _call_impl                                  │
│                                                                                                                     │
│   1526 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                                      │
│ > 1527 │   │   │   return forward_call(*args, **kwargs)                                                             │
│   1528                                                                                                              │
│                                                                                                                     │
│ H:\AI\sd_next\extensions-builtin\sd-webui-controlnet\scripts\cldm.py:31 in forward                                  │
│                                                                                                                     │
│    30 │   def forward(self, *args, **kwargs):                                                                       │
│ >  31 │   │   return self.control_model(*args, **kwargs)                                                            │
│    32                                                                                                               │
│                                                                                                                     │
│ H:\AI\sd_next\venv\lib\site-packages\torch\nn\modules\module.py:1518 in _wrapped_call_impl                          │
│                                                                                                                     │
│   1517 │   │   else:                                                                                                │
│ > 1518 │   │   │   return self._call_impl(*args, **kwargs)                                                          │
│   1519                                                                                                              │
│                                                                                                                     │
│ H:\AI\sd_next\venv\lib\site-packages\torch\nn\modules\module.py:1527 in _call_impl                                  │
│                                                                                                                     │
│   1526 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                                      │
│ > 1527 │   │   │   return forward_call(*args, **kwargs)                                                             │
│   1528                                                                                                              │
│                                                                                                                     │
│ H:\AI\sd_next\extensions-builtin\sd-webui-controlnet\scripts\cldm.py:311 in forward                                 │
│                                                                                                                     │
│   310 │   │   │   │   h = module(h, emb, context)                                                                   │
│ > 311 │   │   │   │   h += guided_hint                                                                              │
│   312 │   │   │   │   guided_hint = None                                                                            │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
RuntimeError: The size of tensor a (64) must match the size of tensor b (96) at non-singleton dimension 3

Version Platform Description

16:50:48-734636 DEBUG Logger: file=H:\AI\sd_next\sdnext.log level=10 size=0 mode=create 16:50:48-737625 INFO Starting SD.Next 16:50:48-738622 INFO Python 3.10.6 on Windows 16:50:48-863205 INFO Version: app=sd.next updated=2023-10-20 hash=d8bb3496 url=https://github.com/vladmandic/automatic.git/tree/master 16:50:49-242392 INFO Platform: arch=AMD64 cpu=Intel64 Family 6 Model 183 Stepping 1, GenuineIntel system=Windows release=Windows-10-10.0.19045-SP0 python=3.10.6 16:50:49-244385 DEBUG Setting environment tuning 16:50:49-245381 DEBUG Torch overrides: cuda=False rocm=False ipex=False diml=False openvino=False 16:50:49-247374 DEBUG Torch allowed: cuda=True rocm=True ipex=True diml=True openvino=True 16:50:49-250365 INFO nVidia CUDA toolkit detected: nvidia-smi present 16:50:49-343054 DEBUG Repository update time: Fri Oct 20 15:31:51 2023 16:51:05-573545 DEBUG Extensions all: ['adetailer', 'multidiffusion-upscaler-for-automatic1111', 'sd-webui-cd-tuner', 'sd-webui-lora-block-weight', 'sd-webui-openpose-editor', 'sd-webui-stablesr', 'sd-webui-vectorscope-cc', 'sd_webui_SAG', 'stable-diffusion-webui-anti-burn', 'ultimate-upscale-for-automatic1111'] 16:51:05-576535 DEBUG Running extension installer: H:\AI\sd_next\extensions\adetailer\install.py 16:51:06-513072 INFO Extensions enabled: ['clip-interrogator-ext', 'Lora', 'sd-extension-chainner', 'sd-extension-system-info', 'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg', 'adetailer', 'multidiffusion-upscaler-for-automatic1111', 'sd-webui-cd-tuner', 'sd-webui-lora-block-weight', 'sd-webui-openpose-editor', 'sd-webui-stablesr', 'sd-webui-vectorscope-cc', 'sd_webui_SAG', 'stable-diffusion-webui-anti-burn', 'ultimate-upscale-for-automatic1111'] 16:51:06-533006 INFO Command line args: ['--autolaunch', '--debug'] autolaunch=True debug=True 16:51:10-005707 DEBUG Loaded packages: torch=2.1.0+cu121 diffusers=0.21.4 gradio=3.43.2 16:51:10-304165 DEBUG Reading: config.json len=56 16:51:10-308151 INFO Engine: backend=Backend.ORIGINAL compute=cuda mode=no_grad device=cuda cross-optimization="Scaled-Dot-Product" 16:51:10-352005 INFO Device: device=NVIDIA GeForce RTX 3090 n=1 arch=sm_90 cap=(8, 6) cuda=12.1 cudnn=8801 driver=545.84

URL link of the extension

https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111

URL link of the issue reported in the extension repository

https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111/issues/318

Acknowledgements

[X] I have read the above and searched for existing issues

Edit: formatting of errors

vladmandic commented 1 year ago

this is a valid issue and well written, but i'll have to place it in my backlog for the moment as deep-dive into interaction between those two is not something i can do right now.

GodEmperor785 commented 1 year ago

I did more tests and I think I found how to fix this problem, though I'm not sure why it fixes the problem.

Tiled Diffusion extension was removed from built-in extensions so it is in extensions/ directory and ControlNet is still a built-in extension so it is in extensions-builtin/ directory. I moved Tiled Diffusion extension directory from extensions/ to extensions-builtin/ (just copy and paste of multidiffusion-upscaler-for-automatic1111 directory in windows file explorer) and now simple testing scenarios seem to work (upload 512x512 img and try to upscale x2 with both extensions enabled) where it previously failed.

Perhaps tiled diffusion and controlnet somehow expect that they are both in the same extensions directory and work only then?

Maybe a solution to this issue would be to make tiled diffusion a built-in extension again like it was in the past? After fixes by extension team in https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111/issues/318 extension seems to work fully with new SD.Next

vladmandic commented 1 year ago

placing extension into extension-builting is not going to happen just to fix interaction between two extensions. extension should never assume its location, there are well defined methods to do that.

vladmandic commented 8 months ago

i'm closing the issue here as i cannot find a record of reported upstream issue and upstream is the only place where this should be fixed.

vladmandic / automatic