search

pkuliyi2015 / multidiffusion-upscaler-for-automatic1111

Tiled Diffusion and VAE optimize, licensed under CC BY-NC-SA 4.0
Other
4.73k stars 334 forks source link

Region Prompt Control gives error RuntimeError: einsum(): subscript b has size 16 for operand 1 which does not broadcast with previously seen size 8 #320

Closed tomvdp closed 10 months ago

tomvdp commented 11 months ago

RuntimeError: einsum(): subscript b has size 16 for operand 1 which does not broadcast with previously seen size 8 This happens when "Region Prompt Control" is enabled. This is without --xformers.

With --xformers, the runtime error is: RuntimeError: shape '[77, 8, 40]' is invalid for input of size 49280

tomvdp commented 11 months ago

I can't get Region Prompt Control to work.

Here is the full error stack:

                                   [Tiled Diffusion] ControlNet found, support is enabled.

Total progress: 36it [03:59, 2.50s/it] MultiDiffusion hooked into 'Euler a' sampler, Tile size: NonexNone, Tile count: None, Batch size: None, Tile batches: 0 (ext: RegionCtrl, ContrlNet) 38%|██████████████████████████████████████████████████████████▏ | 9/24 [00:00<00:01, 11.01it/s] Error completing request 2.36s/it] Arguments: ('task(zob8mqrzrkonord)', 'x', '', [], 24, 'Euler a', 1, 1, 6, 768, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], <gradio.routes.Request object at 0x000001F0EC5D4EB0>, 0, False, '', 0.8, -1, False, -1, 0, 0, 0, True, 'MultiDiffusion', True, True, 1024, 1024, 96, 96, 48, 4, 'None', 2, False, 10, 1, 1, 64, False, True, False, False, True, 0.4, 0.4, 0.2, 0.2, 'x', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, True, 3072, 192, True, True, True, False, False, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '', False, 7, 100, 'Constant', 0, 'Constant', 0, 4, True, 'MEAN', 'AD', 1, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001F0E3592EC0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001F0E6830220>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001F0E6833D30>, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False, None, None, False, None, None, False, None, None, False, 50) {} Traceback (most recent call last): File "F:\projects\sd2\stable-diffusion-webui\modules\call_queue.py", line 57, in f res = list(func(*args, kwargs)) File "F:\projects\sd2\stable-diffusion-webui\modules\call_queue.py", line 36, in f res = func(*args, *kwargs) File "F:\projects\sd2\stable-diffusion-webui\modules\txt2img.py", line 55, in txt2img processed = processing.process_images(p) File "F:\projects\sd2\stable-diffusion-webui\modules\processing.py", line 732, in process_images res = process_images_inner(p) File "F:\projects\sd2\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack return getattr(processing, '__controlnet_original_process_images_inner')(p, args, kwargs) File "F:\projects\sd2\stable-diffusion-webui\modules\processing.py", line 867, in process_images_inner samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts) File "F:\projects\sd2\stable-diffusion-webui\modules\processing.py", line 1140, in sample samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x)) File "F:\projects\sd2\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in sample samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, extra_params_kwargs)) File "F:\projects\sd2\stable-diffusion-webui\modules\sd_samplers_common.py", line 261, in launch_sampling return func() File "F:\projects\sd2\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, extra_params_kwargs)) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, kwargs) File "F:\projects\sd2\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral denoised = model(x, sigmas[i] * s_in, *extra_args) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "F:\projects\sd2\stable-diffusion-webui\modules\sd_samplers_cfg_denoiser.py", line 169, in forward x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict(cond_in, image_cond_in)) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "F:\projects\sd2\stable-diffusion-webui\extensions\multidiffusion-upscaler-for-automatic1111\tile_utils\utils.py", line 249, in wrapper return fn(args, kwargs) File "F:\projects\sd2\stable-diffusion-webui\extensions\multidiffusion-upscaler-for-automatic1111\tile_methods\multidiffusion.py", line 70, in kdiff_forward return self.sample_one_step(x_in, org_func, repeat_func, custom_func) File "F:\projects\sd2\stable-diffusion-webui\extensions\multidiffusion-upscaler-for-automatic1111\tile_methods\multidiffusion.py", line 187, in sample_one_step x_tile_out = custom_func(x_tile, bbox_id, bbox) File "F:\projects\sd2\stable-diffusion-webui\extensions\multidiffusion-upscaler-for-automatic1111\tile_methods\multidiffusion.py", line 68, in custom_func return self.kdiff_custom_forward(x, sigma_in, cond, bbox_id, bbox, self.sampler_forward) File "F:\projects\sd2\stable-diffusion-webui\extensions\multidiffusion-upscaler-for-automatic1111\tile_utils\utils.py", line 249, in wrapper return fn(*args, kwargs) File "F:\projects\sd2\stable-diffusion-webui\extensions\multidiffusion-upscaler-for-automatic1111\tile_methods\abstractdiffusion.py", line 284, in kdiff_custom_forward return forward_func( File "F:\projects\sd2\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), *kwargs) File "F:\projects\sd2\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps return self.inner_model.apply_model(args, kwargs) File "F:\projects\sd2\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in setattr(resolved_obj, func_path[-1], lambda *args, kwargs: self(*args, *kwargs)) File "F:\projects\sd2\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in call return self.__orig_func(args, kwargs) File "F:\projects\sd2\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model x_recon = self.model(x_noisy, t, cond) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "F:\projects\sd2\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward out = self.diffusion_model(x, t, context=cc) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1538, in _call_impl result = forward_call(args, kwargs) File "F:\projects\sd2\stable-diffusion-webui\modules\sd_unet.py", line 91, in UNetModel_forward return ldm.modules.diffusionmodules.openaimodel.copy_of_UNetModel_forward_for_webui(self, x, timesteps, context, *args, kwargs) File "F:\projects\sd2\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 797, in forward h = module(h, emb, context) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "F:\projects\sd2\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 84, in forward x = layer(x, context) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "F:\projects\sd2\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 334, in forward x = block(x, context=context[i]) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "F:\projects\sd2\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 269, in forward return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint) File "F:\projects\sd2\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 121, in checkpoint return CheckpointFunction.apply(func, len(inputs), args) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\function.py", line 506, in apply return super().apply(args, kwargs) # type: ignore[misc] File "F:\projects\sd2\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 136, in forward output_tensors = ctx.run_function(ctx.input_tensors) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\tomesd\patch.py", line 63, in _forward x = u_c(self.attn2(m_c(self.norm2(x)), context=context)) + x File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, *kwargs) File "F:\projects\sd2\stable-diffusion-webui\modules\sd_hijack_optimizations.py", line 496, in xformers_attention_forward out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=None, op=get_xformers_flash_attention_op(q, k, v)) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\xformers\ops\fmha__init.py", line 192, in memory_efficient_attention return _memory_efficient_attention( File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\xformers\ops\fmha__init__.py", line 290, in _memory_efficient_attention return _memory_efficient_attention_forward( File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\xformers\ops\fmha\init__.py", line 310, in _memory_efficient_attention_forward out, _ = op.apply(inp, needs_gradient=False) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\xformers\ops\fmha\flash.py", line 235, in apply ) = _convert_input_format(inp) File "F:\projects\sd2\stable-diffusion-webui\venv\lib\site-packages\xformers\ops\fmha\flash.py", line 177, in _convert_input_format key=key.reshape([batch * seqlen_kv, num_heads, head_dim_q]), RuntimeError: shape '[77, 8, 40]' is invalid for input of size 49280

Total progress: 45it [04:49, 2.36s/it]

tomvdp commented 11 months ago

Solved!

The prompt (at the top) needs BREAK statements. Add as many BREAK statements minus 1 as you have regions enabled.

leucome commented 9 months ago

I started to get this error too after updating A1111 (1.7) and the Extension to the last... "RuntimeError: einsum(): subscript b has size 16 for operand 1 which does not broadcast with previously seen size 8" Adding BREAK only make it generate for a couple of sample then the error come back.

I was not using any BREAK before. So is there somebody who know what is going on? I tested back a backup of A1111 v1.5.1 with ultimate-upscale master [c99f382b] (the one from this summer or something) This combo work fine with no BREAK. Also If i remember correctly it was working on my 1.6 installation but it is the one I updated.

Edit: If I disable "Pad prompt/negative prompt to be same length" or change "Negative Guidance" then the error behave differently. I was able to generate a couple of images like that but it is not guarantee some prompt still cause the error.