Mikubill / sd-webui-controlnet

WebUI extension for ControlNet
GNU General Public License v3.0
16.73k stars 1.93k forks source link

breaks prompt matrix when enabled #203

Closed machado2 closed 1 year ago

machado2 commented 1 year ago

When I use it on txt2img together with prompt matrix, it breaks at the end when it would plot the large image with the matrix. The individual images rendered fine and where saved, just the final plot with all together that fails.


Error completing request█████████████████████████████████████████████████████████████████| 2/2 [00:06<00:00,  3.01s/it]
Arguments: ('task(hen52ry7863ru4a)', 'car | potato', '', [], 1, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, [], 1, False, 'prompt, keyword', 'keyword1', 'None', 'textual inversion first', True, 'none', 'control_scribble-fp16 [c508311e]', 1, {'image': array([[[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       ...,

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]]], dtype=uint8), 'mask': array([[[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       ...,

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]]], dtype=uint8)}, True, 'Scale to Fit (Inner Fit)', False, False, 64, 64, 64, 1, False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0) {}
Traceback (most recent call last):
  File "D:\stable-diffusion-webui\modules\call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "D:\stable-diffusion-webui\modules\call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "D:\stable-diffusion-webui\modules\txt2img.py", line 53, in txt2img
    processed = modules.scripts.scripts_txt2img.run(p, *args)
  File "D:\stable-diffusion-webui\modules\scripts.py", line 376, in run
    processed = script.run(p, *script_args)
  File "D:\stable-diffusion-webui\scripts\prompt_matrix.py", line 102, in run
    grid = images.image_grid(processed.images, p.batch_size, rows=1 << ((len(prompt_matrix_parts) - 1) // 2))
  File "D:\stable-diffusion-webui\modules\images.py", line 52, in image_grid
    grid.paste(img, box=(i % params.cols * w, i // params.cols * h))
  File "d:\stable-diffusion-webui\venv\lib\site-packages\PIL\Image.py", line 1711, in paste
    raise ValueError(msg)
ValueError: cannot determine region size; use 4-item box
Xyem commented 1 year ago

This also breaks https://github.com/Extraltodeus/test_my_prompt. Hazarding a guess that it is because ControlNet emits an extra image which these scripts aren't expecting.

rethink-studios commented 1 year ago

Same here:

Loading model from cache: control_sd15_depth [fef5e48e] Loading preprocessor: depth 0%| | 0/20 [00:00<?, ?it/s]Warning: StyleAdapter and cfg/guess mode may not works due to non-batch-cond inference 100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:02<00:00, 8.70it/s] 100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:02<00:00, 8.59it/s] 100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 10.11it/s] 100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 10.97it/s] Error completing request0:25, 10.91it/s] Arguments: ('task(n3x62zinvfrfkno)', 'a photo of a cat on a blue|white|yellow bed', 'deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime:1.4), text, close up, cropped, out of frame, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra eyes, extra arms, extra legs, fused fingers, too many fingers, long neck\n', [], 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, [], 2, False, True, False, 0, -1, False, '', 0, True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '', False, False, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, 'LoRA', 'None', 1, 1, None, 'Refresh models', False, True, True, 'depth', 'control_sd15_depth [fef5e48e]', 1, {'image': array([[[138, 111, 88], [138, 112, 85], [138, 111, 84], ..., [140, 109, 78], [141, 109, 80], [138, 109, 82]],

   [[140, 112,  86],
    [139, 111,  85],
    [139, 112,  85],
    ...,
    [140, 109,  81],
    [139, 109,  81],
    [140, 109,  79]],

   [[139, 111,  85],
    [139, 112,  85],
    [139, 111,  85],
    ...,
    [140, 109,  80],
    [140, 110,  80],
    [141, 109,  80]],

   ...,

   [[170,   0,  23],
    [166,   1,  22],
    [161,   3,  20],
    ...,
    [239,  16, 103],
    [237,  16,  98],
    [238,  25,  98]],

   [[167,   4,  18],
    [165,   0,  13],
    [159,   0,   9],
    ...,
    [247,  20, 105],
    [241,  12,  97],
    [230,  13,  83]],

   [[149,  18,  19],
    [158,   5,  10],
    [158,   2,  10],
    ...,
    [242,  18,  96],
    [243,  26,  92],
    [223,  42,  88]]], dtype=uint8), 'mask': array([[[  0,   0,   0, 255],
    [  0,   0,   0, 255],
    [  0,   0,   0, 255],
    ...,
    [  0,   0,   0, 255],
    [  0,   0,   0, 255],
    [  0,   0,   0, 255]],

   [[  0,   0,   0, 255],
    [  0,   0,   0, 255],
    [  0,   0,   0, 255],
    ...,
    [  0,   0,   0, 255],
    [  0,   0,   0, 255],
    [  0,   0,   0, 255]],

   [[  0,   0,   0, 255],
    [  0,   0,   0, 255],
    [  0,   0,   0, 255],
    ...,
    [  0,   0,   0, 255],
    [  0,   0,   0, 255],
    [  0,   0,   0, 255]],

   ...,

   [[  0,   0,   0, 255],
    [  0,   0,   0, 255],
    [  0,   0,   0, 255],
    ...,
    [  0,   0,   0, 255],
    [  0,   0,   0, 255],
    [  0,   0,   0, 255]],

   [[  0,   0,   0, 255],
    [  0,   0,   0, 255],
    [  0,   0,   0, 255],
    ...,
    [  0,   0,   0, 255],
    [  0,   0,   0, 255],
    [  0,   0,   0, 255]],

   [[  0,   0,   0, 255],
    [  0,   0,   0, 255],
    [  0,   0,   0, 255],
    ...,
    [  0,   0,   0, 255],
    [  0,   0,   0, 255],
    [  0,   0,   0, 255]]], dtype=uint8)}, False, 'Just Resize', False, False, 384, 64, 64, 0, 1, False, False, 'none', 'None', 1, None, False, 'Scale to Fit (Inner Fit)', False, False, 64, 64, 64, 0, 1, False, False, 'none', 'None', 1, None, False, 'Scale to Fit (Inner Fit)', False, False, 64, 64, 64, 0, 1, False, False, 1, 0.15, False, 'OUT', ['OUT'], 5, 0, 'Bilinear', False, 'Pooling Max', False, 'Lerp', '', '', False, False, False, False, False, '1:1,1:2,1:2', '0:0,0:0,0:1', '0.2,0.8,0.8', 20, '', False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0, False, False, None, True, None, None, False, None, None, None, 50, False, 4.0, '', 10.0, 'Linear', 3, False, True, 30.0, True, False, False, 0, 0.0, 'Lanczos', 1, True, 0, 0, 0.0001) {}

Traceback (most recent call last): File "D:\stable-diffusion-webui\modules\call_queue.py", line 56, in f res = list(func(*args, *kwargs)) File "D:\stable-diffusion-webui\modules\call_queue.py", line 37, in f res = func(args, *kwargs) File "D:\stable-diffusion-webui\modules\txt2img.py", line 53, in txt2img processed = modules.scripts.scripts_txt2img.run(p, args) File "D:\stable-diffusion-webui\modules\scripts.py", line 376, in run processed = script.run(p, *script_args) File "D:\stable-diffusion-webui\scripts\prompt_matrix.py", line 103, in run grid = images.draw_prompt_matrix(grid, processed.images[0].width, processed.images[1].height, prompt_matrix_parts, margin_size) File "D:\stable-diffusion-webui\modules\images.py", line 236, in draw_prompt_matrix return draw_grid_annotations(im, width, height, hor_texts, ver_texts, margin) File "D:\stable-diffusion-webui\modules\images.py", line 178, in draw_grid_annotations assert cols == len(hor_texts), f'bad number of horizontal texts: {len(hor_texts)}; must be {cols}' AssertionError: bad number of horizontal texts: 2; must be 3

Total progress: 160it [00:44, 10.91it/s]

xarthurx commented 1 year ago

Can confirm the issue.

nijiazhi commented 11 months ago

Can confirm the issue.

so, how to solve the problem?

nijiazhi commented 11 months ago

When I use it on txt2img together with prompt matrix, it breaks at the end when it would plot the large image with the matrix. The individual images rendered fine and where saved, just the final plot with all together that fails.


Error completing request█████████████████████████████████████████████████████████████████| 2/2 [00:06<00:00,  3.01s/it]
Arguments: ('task(hen52ry7863ru4a)', 'car | potato', '', [], 1, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, [], 1, False, 'prompt, keyword', 'keyword1', 'None', 'textual inversion first', True, 'none', 'control_scribble-fp16 [c508311e]', 1, {'image': array([[[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       ...,

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]]], dtype=uint8), 'mask': array([[[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       ...,

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]]], dtype=uint8)}, True, 'Scale to Fit (Inner Fit)', False, False, 64, 64, 64, 1, False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0) {}
Traceback (most recent call last):
  File "D:\stable-diffusion-webui\modules\call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "D:\stable-diffusion-webui\modules\call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "D:\stable-diffusion-webui\modules\txt2img.py", line 53, in txt2img
    processed = modules.scripts.scripts_txt2img.run(p, *args)
  File "D:\stable-diffusion-webui\modules\scripts.py", line 376, in run
    processed = script.run(p, *script_args)
  File "D:\stable-diffusion-webui\scripts\prompt_matrix.py", line 102, in run
    grid = images.image_grid(processed.images, p.batch_size, rows=1 << ((len(prompt_matrix_parts) - 1) // 2))
  File "D:\stable-diffusion-webui\modules\images.py", line 52, in image_grid
    grid.paste(img, box=(i % params.cols * w, i // params.cols * h))
  File "d:\stable-diffusion-webui\venv\lib\site-packages\PIL\Image.py", line 1711, in paste
    raise ValueError(msg)
ValueError: cannot determine region size; use 4-item box

have you fixed the problem?

arjuna-dev commented 9 months ago

Same problem here.

An alternative to achieve that functionality is to use a script with X/Y/Z plot and Prompt S/R

x-FrozenLake-x commented 3 months ago

Can confirm the issue.