[Issue]: Find out why new auto1111 extensions will sometimes not work in SD.NEXT

StableInfo commented 1 year ago

Issue Description

After the roop extension, I discoved that the AnimateDiff extension also does not work in sd.next whereas it was made for the original auto1111 (https://github.com/continue-revolution/sd-webui-animatediff),

I obviousey raised an issue on each extension repo, but I am raising one here to ask you or anyone to find why some extensions like theser are not compatible with sd.next, which is really unfortunate. This causes me to HAVE to keep both versions of the UI, just in case I need to use the Roop OR the AnimateDiff extension. I had already adressed the Roop non compatibility: https://github.com/vladmandic/automatic/issues/1499 And today I am writing this issue more like a "global one".

Version Platform Description

app: SD.next updated: 2023-07-28 hash: 77de9cd0

Relevant log output

0%|                                                                                           | 0/25 [00:00<?, ?it/s]
13:33:11-826116 ERROR    Exception:
13:33:11-828119 ERROR    Arguments: args=('task(rgku0347h779zvq)', 'masterpiece, best quality, building crystalin the
                         nature, eclipse', '(low quality, worst quality:1.3), easynegative, bad-hands-5', [], 25, 3,
                         None, False, False, 1, 1, 10, 6, 0.7, 2, 2770943557.0, -1.0, 0, 0, 0, 512, 768, False, 0.26,
                         1.75, 'R-ESRGAN 4x+ Anime6B', 20, 0, 0, 0.8, '', '', [], 0, False, 'MultiDiffusion', False,
                         True, 1024, 1024, 96, 96, 48, 4, 'None', 2, False, 10, 1, 1, 64, False, False, False, False,
                         False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '',
                         'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False,
                         0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '',
                         'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False,
                         0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '',
                         'Background', 0.2, -1.0, False, 3072, 192, True, True, True, False, False, 7, 100, 'Constant',
                         0, 'Constant', 0, 4, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at
                         0x00000250ECF03D30>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at
                         0x00000250ECF010F0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at
                         0x00000250ECF03340>, True, 0, 16, 8, 'mm_sd_v15.ckpt', False, False, 'positive', 'comma', 0,
                         False, False, '', 0, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False, None,
                         None, False, None, None, False, None, None, False, 50) kwargs={}
13:33:11-860116 ERROR    gradio call: AssertionError
┌───────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────┐
│ D:\vladNEXTGEN\automatic\modules\call_queue.py:34 in f                                                              │
│                                                                                                                     │
│    33 │   │   │   try:                                                                                              │
│ >  34 │   │   │   │   res = func(*args, **kwargs)                                                                   │
│    35 │   │   │   │   progress.record_results(id_task, res)                                                         │
│                                                                                                                     │
│ D:\vladNEXTGEN\automatic\modules\txt2img.py:64 in txt2img                                                           │
│                                                                                                                     │
│   63 │   if processed is None:                                                                                      │
│ > 64 │   │   processed = processing.process_images(p)                                                               │
│   65 │   p.close()                                                                                                  │
│                                                                                                                     │
│                                              ... 23 frames hidden ...                                               │
│                                                                                                                     │
│ D:\vladNEXTGEN\automatic\venv\lib\site-packages\torch\nn\modules\module.py:1501 in _call_impl                       │
│                                                                                                                     │
│   1500 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                                      │
│ > 1501 │   │   │   return forward_call(*args, **kwargs)                                                             │
│   1502 │   │   # Do not call functions when jit is used                                                             │
│                                                                                                                     │
│ D:\vladNEXTGEN\automatic\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py:109 │
│ in forward                                                                                                          │
│                                                                                                                     │
│   108 │   def forward(self, x):                                                                                     │
│ > 109 │   │   assert x.shape[1] == self.channels                                                                    │
│   110 │   │   if self.dims == 3:                                                                                    │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
AssertionError

Acknowledgements

[X] I have read the above and searched for existing issues
[X] I confirm that this is classified correctly and its not an extension or diffusers-specific issue

vladmandic commented 1 year ago

there is no single overarching reason, its just that sdnext and a1111 are quite different nowadays. if extension uses well-defined callbacks and follows all the guidelines, it will work as that is guaranteed to be compatible. but if an extension performs some hook into internal app code, there is no guarantees what will happen.

StableInfo commented 1 year ago

So is there a solution? Like making some guidelines of "how to make your extension compatible with SD.NEXT" and fundamentals differences. ? So I can link up that document to the makers of these extensions (like AnimateDiff extension)? Also welcome back @vladmandic

vladmandic commented 1 year ago

not back yet, but trying not to be completely missing in action :) i did write a while back a list of most common mistakes to avoid when writing an extension (plus some guidelines): https://github.com/vladmandic/automatic/wiki/Extensions

i came with that by analyzing top 20 or so "broken" extensions.

StableInfo commented 1 year ago

WIll make sure to share the link;)

vladmandic / automatic