search

lllyasviel / stable-diffusion-webui-forge

GNU Affero General Public License v3.0
8.06k stars 788 forks source link

[Bug]: Controlllite does not work on certain generation dims #66

Open huchenlei opened 8 months ago

huchenlei commented 8 months ago

Checklist

What happened?

Controlllite reports error when given certain generation dimensions.

Steps to reproduce the problem

  1. Set generation dimension to 1000 x 512. After some testing, width of range 1000 ~ 1023 all cause generation error, while 999 and 1024 can successfully generate.
  2. Set preprocessor canny. Select model kohya_controllllite_xl_canny [2ed264be]
  3. Click generate

What should have happened?

Generate without error.

What browsers do you use to access the UI ?

Google Chrome

Sysinfo

No extra cmd args

Console logs

2024-02-06 11:50:35,860 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE
2024-02-06 11:50:35,863 - ControlNet - DEBUG - Use numpy seed 2811570952.
2024-02-06 11:50:35,864 - ControlNet - INFO - Using preprocessor: canny
2024-02-06 11:50:35,864 - ControlNet - INFO - preprocessor resolution = 512
2024-02-06 11:50:36,083 - ControlNet - INFO - Current ControlNet ControlLLLitePatcher: D:\stable-diffusion-webui-forge\models\ControlNet\kohya_controllllite_xl_canny.safetensors
To load target model SDXLClipModel
Begin to load 1 model
unload clone 1
Moving model(s) has taken 1.74 seconds
136 modules
2024-02-06 11:50:38,496 - ControlNet - INFO - ControlNet Method canny patched.
To load target model SDXL
Begin to load 1 model
unload clone 1
Moving model(s) has taken 3.86 seconds
  0%|                                                                                                                                                                                    | 0/20 [00:00<?, ?it/s] 
*** Error completing request
*** Arguments: ('task(w3n0unedbiec9ho)', <gradio.routes.Request object at 0x0000015F8ABCA380>, '', '', [], 20, 'DPM++ 2M Karras', 1, 1, 7, 512, 1023, False, 0.7, 2, 'Latent', 0, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], 0, False, '', 0.8, -1, False, -1, 0, 0, 0, UiControlNetUnit(input_mode=<InputMode.SIMPLE: 'simple'>, use_preview_as_input=False, batch_image_dir='', batch_mask_dir='', batch_input_gallery=[], batch_mask_gallery=[], generated_image=None, mask_image=None, enabled=True, module='canny', model='kohya_controllllite_xl_canny [2ed264be]', weight=1, image={'image': array([[[154, 167, 137],     
***         [155, 168, 138],
***         [155, 168, 138],
***         ...,
***         [ 58,  50,  48],
***         [ 57,  49,  47],
***         [ 56,  48,  46]],
***
***        [[154, 167, 137],
***         [154, 167, 137],
***         [154, 167, 137],
***         ...,
***         [ 58,  50,  48],
***         [ 57,  49,  47],
***         [ 57,  49,  47]],
***
***        [[153, 166, 136],
***         [153, 166, 136],
***         [153, 166, 136],
***         ...,
***         [ 58,  50,  48],
***         [ 57,  49,  47],
***         [ 57,  49,  47]],
***
***        ...,
***
***        [[217, 207, 197],
***         [217, 207, 197],
***         [216, 206, 196],
***         ...,
***         [252, 251, 247],
***         [252, 251, 247],
***         [252, 251, 247]],
***
***        [[217, 207, 197],
***         [217, 207, 197],
***         [217, 207, 197],
***         ...,
***         [252, 251, 247],
***         [252, 251, 247],
***         [252, 251, 247]],
***
***        [[217, 207, 197],
***         [218, 208, 198],
***         [218, 208, 198],
***         ...,
***         [252, 251, 247],
***         [252, 251, 247],
***         [252, 251, 247]]], dtype=uint8), 'mask': array([[[0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0],
***         ...,
***         [0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0]],
***
***        [[0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0],
***         ...,
***         [0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0]],
***
***        [[0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0],
***         ...,
***         [0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0]],
***
***        ...,
***
***        [[0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0],
***         ...,
***         [0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0]],
***
***        [[0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0],
***         ...,
***         [0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0]],
***
***        [[0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0],
***         ...,
***         [0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0]]], dtype=uint8)}, resize_mode='Crop and Resize', processor_res=512, threshold_a=100, threshold_b=200, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced'), UiControlNetUnit(input_mode=<InputMode.SIMPLE: 'simple'>, use_preview_as_input=False, batch_image_dir='', batch_mask_dir='', batch_input_gallery=[], batch_mask_gallery=[], generated_image=None, mask_image=None, enabled=False, module='None', model='None', weight=1, image=None, resize_mode='Crop and Resize', processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced'), UiControlNetUnit(input_mode=<InputMode.SIMPLE: 'simple'>, use_preview_as_input=False, batch_image_dir='', batch_mask_dir='', batch_input_gallery=[], batch_mask_gallery=[], generated_image=None, mask_image=None, enabled=False, module='None', model='None', weight=1, image=None, resize_mode='Crop and Resize', processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced'), UiControlNetUnit(input_mode=<InputMode.SIMPLE: 'simple'>, use_preview_as_input=False, batch_image_dir='', batch_mask_dir='', batch_input_gallery=[], batch_mask_gallery=[], generated_image=None, mask_image=None, enabled=False, module='None', model='None', weight=1, image=None, resize_mode='Crop and Resize', processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced'), UiControlNetUnit(input_mode=<InputMode.SIMPLE: 'simple'>, use_preview_as_input=False, batch_image_dir='', batch_mask_dir='', batch_input_gallery=[], batch_mask_gallery=[], generated_image=None, mask_image=None, enabled=False, module='None', model='None', weight=1, image=None, resize_mode='Crop and Resize', processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced'), UiControlNetUnit(input_mode=<InputMode.SIMPLE: 'simple'>, use_preview_as_input=False, batch_image_dir='', batch_mask_dir='', batch_input_gallery=[], batch_mask_gallery=[], generated_image=None, mask_image=None, enabled=False, module='None', model='None', weight=1, image=None, resize_mode='Crop and Resize', processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced'), False, 1.01, 1.02, 0.99, 0.95, False, 256, 2, 0, False, False, 3, 2, 0, 0.35, True, 'bicubic', 'bicubic', False, 0.5, 2, False, False, False, 'positive', 'comma', 0, False, False, 'start', '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, False, False, False, 0, False) {}
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui-forge\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "D:\stable-diffusion-webui-forge\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "D:\stable-diffusion-webui-forge\modules\txt2img.py", line 110, in txt2img
        processed = processing.process_images(p)
      File "D:\stable-diffusion-webui-forge\modules\processing.py", line 749, in process_images
        res = process_images_inner(p)
      File "D:\stable-diffusion-webui-forge\modules\processing.py", line 920, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "D:\stable-diffusion-webui-forge\modules\processing.py", line 1275, in sample
        samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
      File "D:\stable-diffusion-webui-forge\modules\sd_samplers_kdiffusion.py", line 251, in sample
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "D:\stable-diffusion-webui-forge\modules\sd_samplers_common.py", line 260, in launch_sampling
        return func()
      File "D:\stable-diffusion-webui-forge\modules\sd_samplers_kdiffusion.py", line 251, in <lambda>
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\stable-diffusion-webui-forge\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m
        denoised = model(x, sigmas[i] * s_in, **extra_args)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "D:\stable-diffusion-webui-forge\modules\sd_samplers_cfg_denoiser.py", line 179, in forward
        denoised = forge_sampler.forge_sample(self, denoiser_params=denoiser_params,
      File "D:\stable-diffusion-webui-forge\modules_forge\forge_sampler.py", line 82, in forge_sample
        denoised = sampling_function(model, x, timestep, uncond, cond, cond_scale, model_options, seed)
      File "D:\stable-diffusion-webui-forge\ldm_patched\modules\samplers.py", line 282, in sampling_function
        cond_pred, uncond_pred = calc_cond_uncond_batch(model, cond, uncond_, x, timestep, model_options)
      File "D:\stable-diffusion-webui-forge\ldm_patched\modules\samplers.py", line 253, in calc_cond_uncond_batch
        output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
      File "D:\stable-diffusion-webui-forge\ldm_patched\modules\model_base.py", line 85, in apply_model
        model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "D:\stable-diffusion-webui-forge\ldm_patched\ldm\modules\diffusionmodules\openaimodel.py", line 860, in forward
        h = forward_timestep_embed(module, h, emb, context, transformer_options, time_context=time_context, num_video_frames=num_video_frames, image_only_indicator=image_only_indicator)
      File "D:\stable-diffusion-webui-forge\ldm_patched\ldm\modules\diffusionmodules\openaimodel.py", line 48, in forward_timestep_embed
        x = layer(x, context, transformer_options)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "D:\stable-diffusion-webui-forge\ldm_patched\ldm\modules\attention.py", line 613, in forward
        x = block(x, context=context[i], transformer_options=transformer_options)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "D:\stable-diffusion-webui-forge\ldm_patched\ldm\modules\attention.py", line 440, in forward
        return checkpoint(self._forward, (x, context, transformer_options), self.parameters(), self.checkpoint)
      File "D:\stable-diffusion-webui-forge\ldm_patched\ldm\modules\diffusionmodules\util.py", line 189, in checkpoint
        return func(*inputs)
      File "D:\stable-diffusion-webui-forge\ldm_patched\ldm\modules\attention.py", line 479, in _forward
        n, context_attn1, value_attn1 = p(n, context_attn1, value_attn1, extra_options)
      File "D:\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlllite\lib_controllllite\lib_controllllite.py", line 102, in __call__
        q = q + self.modules[module_pfx_to_q](q)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "D:\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlllite\lib_controllllite\lib_controllllite.py", line 234, in forward
        cx = torch.cat([cx, self.down(x)], dim=1 if self.is_conv2d else 2)
    RuntimeError: Sizes of tensors must match except in dimension 2. Expected size 2016 but got size 2048 for tensor number 1 in the list.

Additional information

No response

huchenlei commented 8 months ago

Note: sd-webui-controlnet's impl does not have this issue.

Kexus commented 6 months ago

I'm having this same problem. 1024x1200 and 800x1200 do not work, but 800x1024 and 1024x1024 do