search

vladmandic / automatic

SD.Next: Advanced Implementation Generative Image Models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.77k stars 431 forks source link

[Extension]: Tiled Diffusion plus Hires result in error sizes of tensors must match except in dimension 0 #1411

Open SAC020 opened 1 year ago

SAC020 commented 1 year ago

Issue Description

Trying to use tiled diffusion with highres fix with controlnets

Original image resolution: 512x830 Highres resize to: 640x1037

Standard tiled diffusion parameters (multidiffusion / 96-sized tiles / 48-sized overlap / batch 4)

Getting the error: Sizes of tensors must match except in dimension 0. Expected size 768 but got size 712 for tensor number 1 in the list

Setting batch = 1 works (but is slow)

Disabling highres or setting scale to 1 works (but it's not my intention)

Not sure what to do about it - set different tile sizes? (what size?). Or is it a bug to be fixed?

TY!

Version Platform Description

13:47:53-215681 INFO Starting SD.Next 13:47:53-220146 INFO Python 3.10.11 on Windows 13:47:53-292478 INFO Version: ec6d0548 Fri Jun 9 09:23:32 2023 -0400 13:47:53-807782 INFO Latest published version: eb47acf552e48a951949a8c63f41818bcfba5520 2023-06-13T02:11:22Z 13:47:53-808774 INFO Setting environment tuning 13:47:53-810757 INFO nVidia CUDA toolkit detected 13:47:58-128432 INFO Torch 2.0.1+cu118 13:47:58-167610 INFO Torch backend: nVidia CUDA 11.8 cuDNN 8700 13:47:58-169470 INFO Torch detected GPU: NVIDIA GeForce RTX 3080 VRAM 10239 Arch (8, 6) Cores 68 13:47:58-349736 WARNING Modified files: ['javascript/themes.json'] 13:47:58-397353 INFO Enabled extensions-builtin: ['a1111-sd-webui-lycoris', 'clip-interrogator-ext', 'LDSR', 'Lora', 'multidiffusion-upscaler-for-automatic1111', 'ScuNET', 'sd-dynamic-thresholding', 'sd-extension-aesthetic-scorer', 'sd-extension-steps-animation', 'sd-extension-system-info', 'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'sd-webui-model-converter', 'seed_travel', 'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg', 'SwinIR'] 13:47:58-417825 INFO Enabled extensions: ['openpose-editor', 'sd-webui-3d-open-pose-editor', 'sd-webui-openpose-editor'] 13:47:58-429133 INFO No changes detected: Quick launch active 13:47:58-430621 INFO Enabled extensions-builtin: ['a1111-sd-webui-lycoris', 'clip-interrogator-ext', 'LDSR', 'Lora', 'multidiffusion-upscaler-for-automatic1111', 'ScuNET', 'sd-dynamic-thresholding', 'sd-extension-aesthetic-scorer', 'sd-extension-steps-animation', 'sd-extension-system-info', 'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'sd-webui-model-converter', 'seed_travel', 'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg', 'SwinIR'] 13:47:58-437069 INFO Enabled extensions: ['openpose-editor', 'sd-webui-3d-open-pose-editor', 'sd-webui-openpose-editor']

Acknowledgements

vladmandic commented 1 year ago

its highly recommended that any resolution (original and desired target) is divisible by 8 - just how models work internally. its not strictly enforced, but its highly recommended.

second, you say it works with batch 1 - so it breaks if you set batch size to 2?

and why combine hires fix and tiled diffusion? typically you'd use tilled diffusion instead of hires fix, i've never tested both as they both do the similar thing. yes, i know, its all about user control and do anything you want, just saying that some scenarios are going to lead to issues

and can you start server with --debug flag and port short log (not pages, just a page) leading to error? based on title alone, i cannot pinpoint where it occurred.

SAC020 commented 1 year ago

Does this help? I censored out the prompts

I have tried the closest resolution divisible by 8, doesnt work

I have tried batch = 2, doesnt work

Can't explain why I am using both, I never tested them in parallel on the same seed :) but it did work on other images, it's not the first image I am trying this combo

2023-06-13 15:03:05,177 | sd | DEBUG | sd_models | Model dict loaded: {'ram': {'used': 1.03, 'total': 31.92}, 'gpu': {'used': 1.16, 'total': 10.0}, 'retries': 0, 'oom': 0}
2023-06-13 15:03:05,189 | sd | DEBUG | sd_models | Model config loaded: {'ram': {'used': 1.03, 'total': 31.92}, 'gpu': {'used': 1.16, 'total': 10.0}, 'retries': 0, 'oom': 0}
2023-06-13 15:03:05,606 | sd | DEBUG | sd_models | Model created from config: C:\ai\automatic\configs\v1-inference.yaml
2023-06-13 15:03:05,608 | sd | DEBUG | sd_models | Model weights loading: {'ram': {'used': 1.99, 'total': 31.92}, 'gpu': {'used': 1.16, 'total': 10.0}, 'retries': 0, 'oom': 0}
2023-06-13 15:03:11,271 | sd | DEBUG | sd_models | Model weights loaded: {'ram': {'used': 5.03, 'total': 31.92}, 'gpu': {'used': 1.16, 'total': 10.0}, 'retries': 0, 'oom': 0}
2023-06-13 15:03:12,120 | sd | DEBUG | sd_models | Model weights moved: {'ram': {'used': 3.04, 'total': 31.92}, 'gpu': {'used': 3.2, 'total': 10.0}, 'retries': 0, 'oom': 0}
2023-06-13 15:03:12,128 | sd | INFO | sd_hijack | Applying scaled dot product cross attention optimization
2023-06-13 15:03:12,167 | sd | INFO | textual_inversion | Embeddings: loaded=0 skipped=0
2023-06-13 15:03:12,175 | sd | INFO | sd_models | Model loaded in 7.4s (load=0.3s config=0.1s create=0.4s apply=5.0s vae=0.7s move=0.8s)
2023-06-13 15:03:12,450 | sd | DEBUG | devices | gc: collected=131 device=cuda {'ram': {'used': 3.07, 'total': 31.92}, 'gpu': {'used': 3.2, 'total': 10.0}, 'retries': 0, 'oom': 0}
2023-06-13 15:03:12,452 | sd | INFO | sd_models | Model load finished: {'ram': {'used': 3.07, 'total': 31.92}, 'gpu': {'used': 3.2, 'total': 10.0}, 'retries': 0, 'oom': 0}
2023-06-13 15:03:59,559 | sd | DEBUG | launch | Server alive: True Memory used: 1.08 total: 31.92
2023-06-13 15:04:17,124 | sd | DEBUG | generation_parameters_copypaste | paste prompt: (photojournalism:1.4), (photorealistic:1.4), (front facing:1.4)
Negative prompt: (worst quality:1.4), (low quality:1.4), blurry, [wrong|anatomy], butterface, plastic, unrealistic, doll, figurine, glossy
Steps: 50, Sampler: DPM++ 2S a Karras, CFG scale: 7, Seed: 3921352636, Size: 512x800, Model hash: af220b387c, Model: epicrealism_newCentury, Denoising strength: 0.7, Clip skip: 1, Version: ec6d054, Parser: Full parser, ControlNet 0: "preprocessor: none, model: control_v11p_sd15_openpose [cab727d4], weight: 2, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: ControlNet is more important, preprocessor params: (512, 64, 64)", ControlNet 1: "preprocessor: none, model: control_v11p_sd15_softedge [a8575a2a], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: My prompt is more important, preprocessor params: (512, 64, 64)", ControlNet 5: "preprocessor: seg_ofade20k, model: control_v11p_sd15_seg [e1f51eb9], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: My prompt is more important, preprocessor params: (512, 64, 64)", Hires upscale: 1.4, Hires steps: 20, Hires upscaler: R-ESRGAN 4x+
2023-06-13 15:06:00,337 | sd | DEBUG | launch | Server alive: True Memory used: 1.08 total: 31.92
2023-06-13 15:08:00,299 | sd | DEBUG | launch | Server alive: True Memory used: 1.33 total: 31.92
2023-06-13 15:10:00,290 | sd | DEBUG | launch | Server alive: True Memory used: 1.52 total: 31.92
2023-06-13 15:11:28,133 | sd | DEBUG | devices | gc: collected=24387 device=cuda {'ram': {'used': 1.52, 'total': 31.92}, 'gpu': {'used': 4.56, 'total': 10.0}, 'retries': 0, 'oom': 0}
2023-06-13 15:11:28,134 | sd | DEBUG | txt2img | txt2img: id_task=task(89r2x3f1q04hdmq)|prompt=(photojournalism:1.4), (photorealistic:1.4), (front facing:1.4)|negative_prompt=(worst quality:1.4), (low quality:1.4), blurry, [wrong|anatomy], butterface, plastic, unrealistic, doll, figurine, glossy|prompt_styles=[]|steps=50|sampler_index=3|restore_faces=False|tiling=False|n_iter=1|batch_size=1|cfg_scale=7|clip_skip=1|seed=-1.0|subseed=-1.0|subseed_strength=0|seed_resize_from_h=0|seed_resize_from_w=0|seed_enable_extras=False|height=960|width=592|enable_hr=True|denoising_strength=0.7|hr_scale=1.2|hr_upscaler=R-ESRGAN 4x+|hr_second_pass_steps=20|hr_resize_x=0|hr_resize_y=0|override_settings_texts=[]args=(0, True, 'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 4, 'None', 2, False, 10, 1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 1536, 96, True, True, True, False, False, 7, 100, 'Constant', 0, 'Constant', 0, 4, False, 'x264', 'blend', 10, 0, 0, False, True, True, True, 'intermediate', 'animation', <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001E71B7E6B60>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001E6034275E0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001E71B7E6EC0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001E71B7E6A10>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001E71B865270>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001E71B864310>, False, False, 'positive', 'comma', 0, False, False, '', 7, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False, None, None, False, None, None, False, None, None, False, None, None, False, None, None, False, None, None, False, 50, False, 4.0, '', 10.0, 'Linear', 3, False, 30.0, True, False, False, 0, 0.0, 'Lanczos', 1, True, 0, 0, 0.001, 75, 0.0, False, True)
2023-06-13 15:11:28,644 | sd | DEBUG | devices | gc: collected=0 device=cuda {'ram': {'used': 1.52, 'total': 31.92}, 'gpu': {'used': 4.56, 'total': 10.0}, 'retries': 0, 'oom': 0}
2023-06-13 15:11:36,692 | sd | DEBUG | devices | gc: collected=0 device=cuda {'ram': {'used': 2.36, 'total': 31.92}, 'gpu': {'used': 4.61, 'total': 10.0}, 'retries': 0, 'oom': 0}
2023-06-13 15:11:41,197 | sd | DEBUG | devices | gc: collected=0 device=cuda {'ram': {'used': 2.37, 'total': 31.92}, 'gpu': {'used': 5.3, 'total': 10.0}, 'retries': 0, 'oom': 0}
2023-06-13 15:11:46,696 | sd | DEBUG | scripts | Script process: ['Tiled Diffusion:0.0s', 'Tiled VAE:0.0s', 'Dynamic Thresholding (CFG Scale Fix):0.0s', 'Steps animation:0.0s', 'ControlNet:18.55s', 'OpenPose Editor:0.0s']
2023-06-13 15:11:46,697 | sd | DEBUG | scripts | Script before-process-batch: ['Tiled Diffusion:0.0s', 'Tiled VAE:0.0s', 'Dynamic Thresholding (CFG Scale Fix):0.0s', 'Steps animation:0.0s', 'ControlNet:0.0s', 'OpenPose Editor:0.0s']
2023-06-13 15:11:46,698 | sd | DEBUG | scripts | Script process-batch: ['Tiled Diffusion:0.0s', 'Tiled VAE:0.0s', 'Dynamic Thresholding (CFG Scale Fix):0.0s', 'Steps animation:0.0s', 'ControlNet:0.0s', 'OpenPose Editor:0.0s']
2023-06-13 15:11:59,823 | sd | DEBUG | launch | Server alive: True Memory used: 3.66 total: 31.92
2023-06-13 15:12:59,043 | sd | ERROR | call_queue | Exception: Sizes of tensors must match except in dimension 0. Expected size 768 but got size 576 for tensor number 1 in the list.
2023-06-13 15:12:59,045 | sd | ERROR | call_queue | Arguments: args=('task(89r2x3f1q04hdmq)', '(photojournalism:1.4), (photorealistic:1.4)(front facing:1.4)', '(worst quality:1.4), (low quality:1.4), [wrong|anatomy], butterface, plastic, unrealistic, doll, figurine, glossy', [], 50, 3, False, False, 1, 1, 7, 1, -1.0, -1.0, 0, 0, 0, False, 960, 592, True, 0.7, 1.2, 'R-ESRGAN 4x+', 20, 0, 0, [], 0, True, 'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 4, 'None', 2, False, 10, 1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 1536, 96, True, True, True, False, False, 7, 100, 'Constant', 0, 'Constant', 0, 4, False, 'x264', 'blend', 10, 0, 0, False, True, True, True, 'intermediate', 'animation', <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001E71B7E6B60>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001E6034275E0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001E71B7E6EC0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001E71B7E6A10>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001E71B865270>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001E71B864310>, False, False, 'positive', 'comma', 0, False, False, '', 7, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False, None, None, False, None, None, False, None, None, False, None, None, False, None, None, False, None, None, False, 50, False, 4.0, '', 10.0, 'Linear', 3, False, 30.0, True, False, False, 0, 0.0, 'Lanczos', 1, True, 0, 0, 0.001, 75, 0.0, False, True) kwargs={}
2023-06-13 15:12:59,052 | sd | ERROR | errors | gradio call: RuntimeError
2023-06-13 15:12:59,424 | sd | DEBUG | devices | gc: collected=18233 device=cuda {'ram': {'used': 3.72, 'total': 31.92}, 'gpu': {'used': 6.9, 'total': 10.0}, 'retries': 0, 'oom': 0}
2023-06-13 15:14:00,102 | sd | DEBUG | launch | Server alive: True Memory used: 3.6 total: 31.92
vladmandic commented 1 year ago

censoring prompt is ok. and yes, this is the correct log (you just quoted it badly so it was wrapped all over the place) it doesn't show much, i'll have to try to reproduce. placing in backlog for now, not sure when this will be done.

SAC020 commented 1 year ago

Thank you very much! In the meantime I will study more on tiled diffusion vs hires, I admit I was using it without really knowing what it does but noticing improvements with it

Aptronymist commented 1 year ago

I feel like there should be a visual indicator or notification of the sizes not being divisible by 8, at least to some degree, probably not possible with extensions.