search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.67k stars 418 forks source link

[Extension]: AnimateDiff #2465

Closed derspanier1 closed 10 months ago

derspanier1 commented 11 months ago

Issue Description

Good Day Frens, after life struck and a torrent of misfortune rolled by, i am finally back to some diffusing. Hope everyone is good. Issue: After completely reinstalling SDNext today, i wanted to do some animating, but i get the following error as soon as i activate controlnet:

14:09:46-676036 INFO     AnimateDiff process start.
14:09:46-692027 INFO     Injecting motion module mm_sd_v15_v2.safetensors into SD1.5 UNet middle block.
14:09:46-694026 INFO     Injecting motion module mm_sd_v15_v2.safetensors into SD1.5 UNet input blocks.
14:09:46-696024 INFO     Injecting motion module mm_sd_v15_v2.safetensors into SD1.5 UNet output blocks.
14:09:46-698023 INFO     Setting DDIM alpha.
14:09:46-703021 INFO     Injection finished.
14:09:46-705019 INFO     Hacking loral to support motion lora
14:09:46-707017 INFO     Hacking CFGDenoiser forward function.
14:09:46-708017 INFO     Hacking ControlNet.
purged
Initializing ----------------------------------------   0% -:--:-- 0:00:00
14:09:58-049483 ERROR    Exception: Expected all tensors to be on the same device, but found at least two devices, cpu
                         and cuda:0! (when checking argument for argument weight in method
                         wrapper_CUDA___slow_conv2d_forward)
14:09:58-065106 ERROR    Arguments: args=('task(plhatqmthz8onvo)', 'a person walking down a path in the woods, with a
                         red backpack on his back', '', [], 20, None, None, True, False, False, 1, 1, 6, 6, 0.7, 1,
                         -1.0, -1.0, 0, 0, 0, 512, 512, False, 0.5, 2, 'None', False, 20, 0, 0, 5, 0.8, '', '', [], 0,
                         False, False, 'positive', 'comma', 0, False, False, '', 0, '', [], 0, '', [], 0, '', [],
                         False, True, False, False, False, False, 0, None, None, False, None, None, False, None, None,
                         False, None, None, False, None, None, False, None, None, False, None, None, False, None, None,
                         False, None, None, False, None, None, False, 50,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002A6BC0C7D30>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002A6BC0C72E0>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002A6BC0C58A0>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002A6BC0CD180>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002A6BC0CD5A0>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002A6BC0C5840>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002A6BC0C4970>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002A6BC0CDC90>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002A6BC0C5D50>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002A6BC0CDF90>,
                         False, False, {'ad_model': 'face_yolov8n.pt', 'ad_prompt': '', 'ad_negative_prompt': '',
                         'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1,
                         'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None',
                         'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True,
                         'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False,
                         'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28,
                         'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint':
                         'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False,
                         'ad_sampler': 'UniPC', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1,
                         'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model':
                         'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1,
                         'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()},
                         {'ad_model': 'None', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3,
                         'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0,
                         'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4,
                         'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True,
                         'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False,
                         'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28,
                         'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint':
                         'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False,
                         'ad_sampler': 'UniPC', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1,
                         'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model':
                         'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1,
                         'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, False,
                         'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 4, 'None', 2, False, 10, 1, 1, 64,
                         False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False,
                         0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '',
                         'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False,
                         0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '',
                         'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False,
                         0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 3072, 192, True, True, True,
                         False, True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False,
                         False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '',
                         <scripts.animatediff_ui.AnimateDiffProcess object at 0x000002A6BC0B7010>,
                         'NONE:0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0\nALL:1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1\nINS:1,1,1,1,0,
                         0,0,0,0,0,0,0,0,0,0,0,0\nIND:1,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,0\nINALL:1,1,1,1,1,1,1,0,0,0,0,0,
                         0,0,0,0,0\nMIDD:1,0,0,0,1,1,1,1,1,1,1,1,0,0,0,0,0\nOUTD:1,0,0,0,0,0,0,0,1,1,1,1,0,0,0,0,0\nOUT
                         S:1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1\nOUTALL:1,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1\nALL0.5:0.5,0.5,
                         0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', True, 0, 'values',
                         '0,0.25,0.5,0.75,1', 'Block ID', 'IN05-OUT05', 'none', '', '0.5,1',
                         'BASE,IN00,IN01,IN02,IN03,IN04,IN05,IN06,IN07,IN08,IN09,IN10,IN11,M00,OUT00,OUT01,OUT02,OUT03,
                         OUT04,OUT05,OUT06,OUT07,OUT08,OUT09,OUT10,OUT11', 1.0, 'black', '20', False,
                         'ATTNDEEPON:IN05-OUT05:attn:1\n\nATTNDEEPOFF:IN05-OUT05:attn:0\n\nPROJDEEPOFF:IN05-OUT05:proj:
                         0\n\nXYZ:::1', False, False, None, False, '0', '0', 'inswapper_128.onnx', 'CodeFormer', 1,
                         True, 'None', 1, 1, False, True, 1, 0, 0, False, 0.5, True, False, 'CUDA', False, 0.75, 1,
                         False, False, 9, 1) kwargs={}
14:09:58-165356 ERROR    gradio call: RuntimeError
┌───────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────┐
│ H:\SDNext_Master\modules\call_queue.py:34 in f                                                                      │
│                                                                                                                     │
│   33 │   │   │   try:                                                                                               │
│ > 34 │   │   │   │   res = func(*args, **kwargs)                                                                    │
│   35 │   │   │   │   progress.record_results(id_task, res)                                                          │
│                                                                                                                     │
│ H:\SDNext_Master\modules\txt2img.py:66 in txt2img                                                                   │
│                                                                                                                     │
│   65 │   if processed is None:                                                                                      │
│ > 66 │   │   processed = processing.process_images(p)                                                               │
│   67 │   p.close()                                                                                                  │
│                                                                                                                     │
│ H:\SDNext_Master\modules\processing.py:712 in process_images                                                        │
│                                                                                                                     │
│    711 │   │   │   with context_hypertile_vae(p), context_hypertile_unet(p):                                        │
│ >  712 │   │   │   │   res = process_images_inner(p)                                                                │
│    713 │   finally:                                                                                                 │
│                                                                                                                     │
│ H:\SDNext_Master\extensions\sd-webui-animatediff\scripts\animatediff_cn.py:118 in                                   │
│ hacked_processing_process_images_hijack                                                                             │
│                                                                                                                     │
│   117 │   │   │   update_infotext(p, params)                                                                        │
│ > 118 │   │   │   return getattr(processing, '__controlnet_original_process_images_inner')(p,                       │
│   119                                                                                                               │
│                                                                                                                     │
│ H:\SDNext_Master\modules\processing.py:837 in process_images_inner                                                  │
│                                                                                                                     │
│    836 │   │   │   │   with devices.without_autocast() if devices.unet_needs_upcast else device                     │
│ >  837 │   │   │   │   │   samples_ddim = p.sample(conditioning=c, unconditional_conditioning=u                     │
│    838 │   │   │   │   x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(                     │
│                                                                                                                     │
│                                              ... 34 frames hidden ...                                               │
│                                                                                                                     │
│ H:\SDNext_Master\repositories\ldm\modules\diffusionmodules\openaimodel.py:86 in forward                             │
│                                                                                                                     │
│    85 │   │   │   else:                                                                                             │
│ >  86 │   │   │   │   x = layer(x)                                                                                  │
│    87 │   │   return x                                                                                              │
│                                                                                                                     │
│ H:\SDNext_Master\venv\lib\site-packages\torch\nn\modules\module.py:1518 in _wrapped_call_impl                       │
│                                                                                                                     │
│   1517 │   │   else:                                                                                                │
│ > 1518 │   │   │   return self._call_impl(*args, **kwargs)                                                          │
│   1519                                                                                                              │
│                                                                                                                     │
│ H:\SDNext_Master\venv\lib\site-packages\torch\nn\modules\module.py:1527 in _call_impl                               │
│                                                                                                                     │
│   1526 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                                      │
│ > 1527 │   │   │   return forward_call(*args, **kwargs)                                                             │
│   1528                                                                                                              │
│                                                                                                                     │
│ H:\SDNext_Master\venv\lib\site-packages\torch\nn\modules\conv.py:460 in forward                                     │
│                                                                                                                     │
│    459 │   def forward(self, input: Tensor) -> Tensor:                                                              │
│ >  460 │   │   return self._conv_forward(input, self.weight, self.bias)                                             │
│    461                                                                                                              │
│                                                                                                                     │
│ H:\SDNext_Master\venv\lib\site-packages\torch\nn\modules\conv.py:456 in _conv_forward                               │
│                                                                                                                     │
│    455 │   │   │   │   │   │   │   _pair(0), self.dilation, self.groups)                                            │
│ >  456 │   │   return F.conv2d(input, weight, bias, self.stride,                                                    │
│    457 │   │   │   │   │   │   self.padding, self.dilation, self.groups)                                            │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when
checking argument for argument weight in method wrapper_CUDA___slow_conv2d_forward)

This is far beyond my understanding, and even though i really don´t like to bother, i have no clue what to do.

Version Platform Description

Starting SD.Next
Python 3.10.11 on Windows
Version: app=sd.next updated=2023-11-08 hash=a03d7ce9
url=https://github.com/vladmandic/automatic/tree/master
Platform: arch=AMD64 cpu=Intel64 Family 6 Model 60 Stepping 3, GenuineIntel system=Windows
release=Windows-10-10.0.19045-SP0 python=3.10.11
nVidia CUDA toolkit detected: nvidia-smi present
Startup: standard
Verifying requirements
Verifying packages
Verifying submodules
Extensions enabled: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info',
'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser',
'stable-diffusion-webui-rembg', 'a1111-sd-webui-tagcomplete', 'adetailer',
'multidiffusion-upscaler-for-automatic1111', 'sd-dynamic-prompts', 'sd-webui-animatediff',
'sd-webui-lora-block-weight', 'sd-webui-openpose-editor', 'sd-webui-photopea-embed',
'sd-webui-reactor', 'sd_webui_SAG', 'ultimate-upscale-for-automatic1111']
Verifying requirements
Updating Wiki
Extension preload: {'extensions-builtin': 0.01, 'extensions': 0.0}
Command line args: ['--upgrade'] upgrade=True
Load packages: torch=2.1.0+cu121 diffusers=0.22.0 gradio=3.43.2
Engine: backend=Backend.ORIGINAL compute=cuda mode=no_grad device=cuda
cross-optimization="Scaled-Dot-Product"
Device: device=NVIDIA GeForce RTX 3090 n=1 arch=sm_90 cap=(8, 6) cuda=12.1 cudnn=8801
driver=546.01
Available VAEs: path="models\VAE" items=13
Disabling uncompatible extensions: backend=Backend.ORIGINAL []
Available models: path="D:\Stable-diffusion" items=51 time=0.05
Extension: script='extensions-builtin\sd-webui-agent-scheduler\scripts\task_scheduler.py'
Using sqlite file: extensions-builtin\sd-webui-agent-scheduler\task_scheduler.sqlite3
Extension: script='extensions-builtin\sd-webui-controlnet\scripts\controlnet.py' Warning:
ControlNet failed to load SGM - will use LDM instead.
Extension: script='extensions-builtin\sd-webui-controlnet\scripts\controlnet.py' ControlNet
preprocessor location:
H:\SDNext_Master\extensions-builtin\sd-webui-controlnet\annotator\downloads
Extension: script='extensions-builtin\sd-webui-controlnet\scripts\hook.py' Warning: ControlNet
failed to load SGM - will use LDM instead.
Extension: script='extensions\a1111-sd-webui-tagcomplete\scripts\tag_autocomplete_helper.py'
Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will
be limited to those added through the extra networks menu.
Extension: script='extensions\adetailer\scripts\!adetailer.py' [-] ADetailer initialized.
version: 23.11.0, num models: 13
Extensions time: 5.98 { Lora=1.26 sd-extension-chainner=0.12 sd-extension-system-info=0.07
sd-webui-agent-scheduler=0.53 sd-webui-controlnet=0.57
stable-diffusion-webui-images-browser=0.18 stable-diffusion-webui-rembg=1.15
a1111-sd-webui-tagcomplete=0.32 adetailer=0.99 sd-webui-animatediff=0.22
sd-webui-openpose-editor=0.07 sd-webui-photopea-embed=0.07 sd-webui-reactor=0.38 }
Load UI theme: name="black-teal" style=Auto base=style.css
Local URL: http://127.0.0.1:7860/
Initializing middleware
[AgentScheduler] Task queue is empty
[AgentScheduler] Registering APIs
Select: model="Artistic\realcartoon3d_v8 [4e089bea6e]"
Loading weights: D:\Stable-diffusion\Artistic\realcartoon3d_v8.safetensors ------------------------- 0.0/2.1 GB -:--:--
Torch override VAE dtype: no-half set
Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float32 unet=torch.float16
context=no_grad fp16=True bf16=False
LDM: LatentDiffusion: Running in eps-prediction mode
LDM: DiffusionWrapper has 859.52 M params.
LDM: LatentDiffusion: Running in eps-prediction mode
LDM: DiffusionWrapper has 859.52 M params.
Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline
file="D:\Stable-diffusion\Artistic\realcartoon3d_v8.safetensors" size=2034MB
Loading weights: models\VAE\vae-ft-mse-840000-ema-pruned.safetensors ----------------------------- 0.0/334.6 MB -:--:--
Cross-attention: optimization=Scaled-Dot-Product options=[]
Load embeddings: loaded=256 skipped=12 time=0.63
Model loaded in 11.62 { load=0.05 config=0.07 create=7.92 apply=0.95 vae=1.17 move=0.81
embeddings=0.63 }
Model load finished: {'ram': {'used': 4.49, 'total': 15.95}, 'gpu': {'used': 3.47, 'total':
24.0}, 'retries': 0, 'oom': 0} cached=0
Startup time: 34.03 { torch=6.59 gradio=0.58 diffusers=0.05 libraries=1.54 extensions=5.98
models=0.05 face-restore=0.23 upscalers=0.20 ui-extra-networks=0.96 ui-txt2img=0.29
ui-img2img=0.33 ui-settings=0.50 ui-extensions=2.48 ui-defaults=0.18 launch=0.58 api=0.11
app-started=0.99 checkpoint=12.26 }
MOTD: N/A
Browser session: client=127.0.0.1 agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0)
Gecko/20100101 Firefox/119.0

URL link of the extension

https://github.com/continue-revolution/sd-webui-animatediff

URL link of the issue reported in the extension repository

https://github.com/continue-revolution/sd-webui-animatediff/issues/292

Acknowledgements

vladmandic commented 11 months ago

the root cause of this error is if two models are used, but one model is on gpu and another on cpu. and each time, it needs to be tracked down to which exact models - but in this case, i cannot reproduce.

but why are you using vae-ft-mse-840000-ema-pruned AND setting no-half? those are pretty much obsolete unless

derspanier1 commented 11 months ago

so could it be that the vae gets loaded into ram ( CPU ) ? because i see the ram filling up and i do not understand why. maybe the motion module ? i removed no-half, and the "correct" setting for vae would be "none" ? ( give the model used is "good" and has vae baked in ?

derspanier1 commented 11 months ago

okay: i turned off no half, and set vae to none, and i get the same error as soon as i turn on controlnet, no matter if i provide a single image or batch a folder with an image sequence. when i observe the task manager, it looks like the motion module of animatediff gets loaded into the ram. can that be the problem ?

but i realize now that it could also be controlnet being loaded into ram.

just for completeness, i was following this tut, and for him ( using a1111) it works with a single image and batch and additional video guidance: https://youtu.be/s8sUJT1AngI?si=e8vvERIvo3K39_Wl&t=252

vladmandic commented 11 months ago

ah, you're using animatediff WITH controlnet - that was not noted in the original issue. try latest version, i just did some updates. if it still doesn't work, post a screenshot of your animatediff settings.

re: vae - yes, setting it to none and without no-half is fine.

derspanier1 commented 11 months ago

feedback: Single image and batch and additional video driver seems to be golden. But i get the following error without controlnet activated:

17:51:40-832581 ERROR    Error running before process: extensions\sd-webui-animatediff\scripts\animatediff.py:
                         AttributeError
┌───────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────┐
│ H:\SDNext_Master\modules\scripts.py:508 in before_process                                                           │
│                                                                                                                     │
│   507 │   │   │   │   script_args = p.script_args[script.args_from:script.args_to]                                  │
│ > 508 │   │   │   │   script.before_process(p, *script_args, **kwargs)                                              │
│   509 │   │   │   except Exception as e:                                                                            │
│                                                                                                                     │
│ H:\SDNext_Master\extensions\sd-webui-animatediff\scripts\animatediff.py:50 in before_process                        │
│                                                                                                                     │
│    49 │   │   │   params.set_p(p)                                                                                   │
│ >  50 │   │   │   motion_module.inject(p.sd_model, params.model)                                                    │
│    51 │   │   │   self.prompt_scheduler = AnimateDiffPromptSchedule()                                               │
│                                                                                                                     │
│ H:\SDNext_Master\extensions\sd-webui-animatediff\scripts\animatediff_mm.py:59 in inject                             │
│                                                                                                                     │
│    58 │   │   self._load(model_name)                                                                                │
│ >  59 │   │   inject_sdxl = sd_model.is_sdxl or self.mm.is_sdxl                                                     │
│    60 │   │   sd_ver = "SDXL" if sd_model.is_sdxl else "SD1.5"                                                      │
│                                                                                                                     │
│ H:\SDNext_Master\venv\lib\site-packages\torch\nn\modules\module.py:1695 in __getattr__                              │
│                                                                                                                     │
│   1694 │   │   │   │   return modules[name]                                                                         │
│ > 1695 │   │   raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'"                     │
│   1696                                                                                                              │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
AttributeError: 'LatentDiffusion' object has no attribute 'is_sdxl'

this error has the effect that there seems to be no "consistency" anymore from one frame to the next. i will do further testing after eating a quick dinner. brb

vladmandic commented 11 months ago

i need EXACT workflow to reproduce. with standard workflow, it works fine. so please screenshot of your animatediff settings (i did say that before).

derspanier1 commented 11 months ago

Sorry ! i saw there was another update, and now the previous error is back :/ Here the screenshot, animatediff settings on the bottom: New Project (23)

ill test further

derspanier1 commented 11 months ago

clean start, settings as above, frames folder has correctly numbered jpgs ( walkin4_00000.jpg, walkin4_00001.jpg and so on) i click generate:

EDIT: yes you are right, standard workflow works

Using VENV: H:\SDNext_Master\venv
18:47:23-711156 INFO     Starting SD.Next
18:47:23-711156 INFO     Python 3.10.11 on Windows
18:47:23-842655 INFO     Version: app=sd.next updated=2023-11-08 hash=3c9c9cf4
                         url=https://github.com/vladmandic/automatic/tree/master
18:47:24-112159 INFO     Platform: arch=AMD64 cpu=Intel64 Family 6 Model 60 Stepping 3, GenuineIntel system=Windows
                         release=Windows-10-10.0.19045-SP0 python=3.10.11
18:47:24-112159 INFO     nVidia CUDA toolkit detected: nvidia-smi present
18:47:24-196784 INFO     Startup: standard
18:47:24-196784 INFO     Verifying requirements
18:47:24-212411 INFO     Verifying packages
18:47:24-212411 INFO     Verifying submodules
18:47:53-177627 INFO     Extensions enabled: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser',
                         'stable-diffusion-webui-rembg', 'a1111-sd-webui-tagcomplete', 'adetailer',
                         'multidiffusion-upscaler-for-automatic1111', 'sd-dynamic-prompts', 'sd-webui-animatediff',
                         'sd-webui-lora-block-weight', 'sd-webui-openpose-editor', 'sd-webui-photopea-embed',
                         'sd-webui-reactor', 'sd_webui_SAG', 'ultimate-upscale-for-automatic1111']
18:47:53-193252 INFO     Verifying requirements
18:47:53-224501 INFO     Updating Wiki
18:47:53-995267 INFO     Extension preload: {'extensions-builtin': 0.0, 'extensions': 0.0}
18:47:53-995267 INFO     Command line args: ['--upgrade'] upgrade=True
18:48:00-957746 INFO     Load packages: torch=2.1.0+cu121 diffusers=0.22.3 gradio=3.43.2
18:48:01-697265 INFO     Engine: backend=Backend.ORIGINAL compute=cuda mode=no_grad device=cuda
                         cross-optimization="Scaled-Dot-Product"
18:48:01-744142 INFO     Device: device=NVIDIA GeForce RTX 3090 n=1 arch=sm_90 cap=(8, 6) cuda=12.1 cudnn=8801
                         driver=546.01
18:48:02-499431 INFO     Available VAEs: path="models\VAE" items=13
18:48:02-501444 INFO     Disabling uncompatible extensions: backend=Backend.ORIGINAL []
18:48:02-541628 INFO     Available models: path="D:\Stable-diffusion" items=51 time=0.04
18:48:04-911048 INFO     Extension: script='extensions-builtin\sd-webui-agent-scheduler\scripts\task_scheduler.py'
                         Using sqlite file: extensions-builtin\sd-webui-agent-scheduler\task_scheduler.sqlite3
18:48:05-540226 INFO     Extension: script='extensions-builtin\sd-webui-controlnet\scripts\controlnet.py' Warning:
                         ControlNet failed to load SGM - will use LDM instead.
18:48:05-543406 INFO     Extension: script='extensions-builtin\sd-webui-controlnet\scripts\controlnet.py' ControlNet
                         preprocessor location:
                         H:\SDNext_Master\extensions-builtin\sd-webui-controlnet\annotator\downloads
18:48:05-556756 INFO     Extension: script='extensions-builtin\sd-webui-controlnet\scripts\hook.py' Warning: ControlNet
                         failed to load SGM - will use LDM instead.
18:48:07-362263 INFO     Extension: script='extensions\a1111-sd-webui-tagcomplete\scripts\tag_autocomplete_helper.py'
                         Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will
                         be limited to those added through the extra networks menu.
18:48:08-413583 INFO     Extension: script='extensions\adetailer\scripts\!adetailer.py' [-] ADetailer initialized.
                         version: 23.11.0, num models: 13
18:48:09-301474 INFO     Extensions time: 6.51 { Lora=1.34 sd-extension-chainner=0.10 sd-extension-system-info=0.09
                         sd-webui-agent-scheduler=0.57 sd-webui-controlnet=0.66
                         stable-diffusion-webui-images-browser=0.20 stable-diffusion-webui-rembg=1.25
                         a1111-sd-webui-tagcomplete=0.35 adetailer=1.06 sd-webui-animatediff=0.24
                         sd-webui-openpose-editor=0.10 sd-webui-photopea-embed=0.08 sd-webui-reactor=0.39 }
18:48:09-672340 INFO     Load UI theme: name="black-teal" style=Auto base=style.css
18:48:14-973758 INFO     Local URL: http://127.0.0.1:7860/
18:48:14-973758 INFO     Initializing middleware
18:48:15-325872 INFO     [AgentScheduler] Task queue is empty
18:48:15-333894 INFO     [AgentScheduler] Registering APIs
18:48:16-103067 INFO     Select: model="Artistic\realcartoon3d_v8 [4e089bea6e]"
Loading weights: D:\Stable-diffusion\Artistic\realcartoon3d_v8.safetensors ------------------------- 0.0/2.1 GB -:--:--
18:48:16-200481 INFO     Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16
                         context=no_grad fp16=True bf16=False
18:48:25-189295 INFO     LDM: LatentDiffusion: Running in eps-prediction mode
18:48:25-189295 INFO     LDM: DiffusionWrapper has 859.52 M params.
18:48:25-189295 INFO     LDM: LatentDiffusion: Running in eps-prediction mode
18:48:25-189295 INFO     LDM: DiffusionWrapper has 859.52 M params.
18:48:25-204919 INFO     Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline
                         file="D:\Stable-diffusion\Artistic\realcartoon3d_v8.safetensors" size=2034MB
18:48:35-404897 INFO     Cross-attention: optimization=Scaled-Dot-Product options=[]
18:48:36-156511 INFO     Load embeddings: loaded=256 skipped=12 time=0.74
18:48:36-164521 INFO     Model loaded in 20.06 { load=0.05 config=0.06 create=8.99 apply=9.16 vae=0.49 move=0.55
                         embeddings=0.74 }
18:48:36-897939 INFO     Model load finished: {'ram': {'used': 4.08, 'total': 15.95}, 'gpu': {'used': 3.31, 'total':
                         24.0}, 'retries': 0, 'oom': 0} cached=0
18:48:37-023823 INFO     Startup time: 43.01 { torch=6.31 gradio=0.60 libraries=1.53 extensions=6.51 models=0.05
                         face-restore=0.24 upscalers=0.22 ui-extra-networks=0.99 ui-txt2img=0.28 ui-img2img=0.34
                         ui-settings=0.51 ui-extensions=2.42 ui-defaults=0.16 launch=0.62 api=0.11 app-started=1.02
                         checkpoint=20.92 }
18:48:39-393543 INFO     MOTD: N/A
18:48:51-888053 INFO     Browser session: client=127.0.0.1 agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0)
                         Gecko/20100101 Firefox/119.0
18:54:31-127540 INFO     AnimateDiff process start.
18:54:31-130537 INFO     Loading motion module mm_sd_v15_v2.safetensors from
                         extensions\sd-webui-animatediff\model\mm_sd_v15_v2.safetensors
Loading weights: extensions\sd-webui-animatediff\model\mm_sd_v15_v2.safetensors -------------------- 0.0/1.8 GB -:--:--
18:54:31-292445 INFO     Guessed mm_sd_v15_v2.safetensors architecture: MotionModuleType.AnimateDiffV2
18:54:38-581852 WARNING  Missing keys <All keys matched successfully>
18:54:39-133322 INFO     Injecting motion module mm_sd_v15_v2.safetensors into SD1.5 UNet middle block.
18:54:39-133322 INFO     Injecting motion module mm_sd_v15_v2.safetensors into SD1.5 UNet input blocks.
18:54:39-133322 INFO     Injecting motion module mm_sd_v15_v2.safetensors into SD1.5 UNet output blocks.
18:54:39-133322 INFO     Setting DDIM alpha.
18:54:39-164572 INFO     Injection finished.
18:54:39-164572 INFO     Hacking loral to support motion lora
18:54:39-164572 INFO     Hacking CFGDenoiser forward function.
18:54:39-180197 INFO     Hacking ControlNet.
purged
  0%|                                                                                           | 0/30 [00:00<?, ?it/s]
18:55:26-154509 ERROR    Exception: Expected all tensors to be on the same device, but found at least two devices, cpu
                         and cuda:0! (when checking argument for argument weight in method
                         wrapper_CUDA___slow_conv2d_forward)
18:55:26-154509 ERROR    Arguments: args=('task(mamkjeoriu887ut)', '1girl with long hair in the wind, wearing a red
                         backpack, walking down a path in the woods, bright fence on the right side, BREAK,', '', [],
                         30, 7, None, True, False, False, 1, 1, 6, 6, 0.7, 1, -1.0, -1.0, 0, 0, 0, 512, 512, False,
                         0.5, 2, 'None', False, 20, 0, 0, 5, 0.8, '', '', [], 0, False, False, 'positive', 'comma', 0,
                         False, False, '', 0, '', [], 0, '', [], 0, '', [], False, True, False, False, False, False, 0,
                         None, None, False, None, None, False, None, None, False, None, None, False, None, None, False,
                         None, None, False, None, None, False, None, None, False, None, None, False, None, None, False,
                         50, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000024C3B88BAC0>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000024C3B855DB0>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000024C3B857A90>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000024C3B855DE0>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000024C3E4F7AC0>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000024C3E745180>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000024C3B854CD0>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000024C3B855D50>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000024C3E4E5990>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000024C3AEA4E20>,
                         False, False, {'ad_model': 'face_yolov8n.pt', 'ad_prompt': '', 'ad_negative_prompt': '',
                         'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1,
                         'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None',
                         'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True,
                         'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False,
                         'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28,
                         'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint':
                         'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False,
                         'ad_sampler': 'UniPC', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1,
                         'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model':
                         'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1,
                         'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()},
                         {'ad_model': 'None', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3,
                         'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0,
                         'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4,
                         'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True,
                         'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False,
                         'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28,
                         'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint':
                         'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False,
                         'ad_sampler': 'UniPC', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1,
                         'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model':
                         'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1,
                         'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, False,
                         'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 4, 'None', 2, False, 10, 1, 1, 64,
                         False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False,
                         0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '',
                         'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False,
                         0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '',
                         'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False,
                         0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 3072, 192, True, True, True,
                         False, True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False,
                         False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '',
                         <scripts.animatediff_ui.AnimateDiffProcess object at 0x0000024C3E4E6F50>,
                         'NONE:0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0\nALL:1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1\nINS:1,1,1,1,0,
                         0,0,0,0,0,0,0,0,0,0,0,0\nIND:1,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,0\nINALL:1,1,1,1,1,1,1,0,0,0,0,0,
                         0,0,0,0,0\nMIDD:1,0,0,0,1,1,1,1,1,1,1,1,0,0,0,0,0\nOUTD:1,0,0,0,0,0,0,0,1,1,1,1,0,0,0,0,0\nOUT
                         S:1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1\nOUTALL:1,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1\nALL0.5:0.5,0.5,
                         0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', True, 0, 'values',
                         '0,0.25,0.5,0.75,1', 'Block ID', 'IN05-OUT05', 'none', '', '0.5,1',
                         'BASE,IN00,IN01,IN02,IN03,IN04,IN05,IN06,IN07,IN08,IN09,IN10,IN11,M00,OUT00,OUT01,OUT02,OUT03,
                         OUT04,OUT05,OUT06,OUT07,OUT08,OUT09,OUT10,OUT11', 1.0, 'black', '20', False,
                         'ATTNDEEPON:IN05-OUT05:attn:1\n\nATTNDEEPOFF:IN05-OUT05:attn:0\n\nPROJDEEPOFF:IN05-OUT05:proj:
                         0\n\nXYZ:::1', False, False, None, False, '0', '0', 'inswapper_128.onnx', 'CodeFormer', 1,
                         True, 'None', 1, 1, False, True, 1, 0, 0, False, 0.5, True, False, 'CUDA', False, 0.75, 1,
                         False, False, 9, 1) kwargs={}
18:55:26-270388 ERROR    gradio call: RuntimeError
┌───────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────┐
│ H:\SDNext_Master\modules\call_queue.py:34 in f                                                                      │
│                                                                                                                     │
│   33 │   │   │   try:                                                                                               │
│ > 34 │   │   │   │   res = func(*args, **kwargs)                                                                    │
│   35 │   │   │   │   progress.record_results(id_task, res)                                                          │
│                                                                                                                     │
│ H:\SDNext_Master\modules\txt2img.py:66 in txt2img                                                                   │
│                                                                                                                     │
│   65 │   if processed is None:                                                                                      │
│ > 66 │   │   processed = processing.process_images(p)                                                               │
│   67 │   p.close()                                                                                                  │
│                                                                                                                     │
│ H:\SDNext_Master\modules\processing.py:742 in process_images                                                        │
│                                                                                                                     │
│    741 │   │   │   with context_hypertile_vae(p), context_hypertile_unet(p):                                        │
│ >  742 │   │   │   │   res = process_images_inner(p)                                                                │
│    743 │   finally:                                                                                                 │
│                                                                                                                     │
│ H:\SDNext_Master\extensions\sd-webui-animatediff\scripts\animatediff_cn.py:118 in                                   │
│ hacked_processing_process_images_hijack                                                                             │
│                                                                                                                     │
│   117 │   │   │   update_infotext(p, params)                                                                        │
│ > 118 │   │   │   return getattr(processing, '__controlnet_original_process_images_inner')(p,                       │
│   119                                                                                                               │
│                                                                                                                     │
│ H:\SDNext_Master\modules\processing.py:873 in process_images_inner                                                  │
│                                                                                                                     │
│    872 │   │   │   │   with devices.without_autocast() if devices.unet_needs_upcast else device                     │
│ >  873 │   │   │   │   │   samples_ddim = p.sample(conditioning=c, unconditional_conditioning=u                     │
│    874 │   │   │   │   x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(                     │
│                                                                                                                     │
│                                              ... 33 frames hidden ...                                               │
│                                                                                                                     │
│ H:\SDNext_Master\repositories\ldm\modules\diffusionmodules\openaimodel.py:86 in forward                             │
│                                                                                                                     │
│    85 │   │   │   else:                                                                                             │
│ >  86 │   │   │   │   x = layer(x)                                                                                  │
│    87 │   │   return x                                                                                              │
│                                                                                                                     │
│ H:\SDNext_Master\venv\lib\site-packages\torch\nn\modules\module.py:1518 in _wrapped_call_impl                       │
│                                                                                                                     │
│   1517 │   │   else:                                                                                                │
│ > 1518 │   │   │   return self._call_impl(*args, **kwargs)                                                          │
│   1519                                                                                                              │
│                                                                                                                     │
│ H:\SDNext_Master\venv\lib\site-packages\torch\nn\modules\module.py:1527 in _call_impl                               │
│                                                                                                                     │
│   1526 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                                      │
│ > 1527 │   │   │   return forward_call(*args, **kwargs)                                                             │
│   1528                                                                                                              │
│                                                                                                                     │
│ H:\SDNext_Master\venv\lib\site-packages\torch\nn\modules\conv.py:460 in forward                                     │
│                                                                                                                     │
│    459 │   def forward(self, input: Tensor) -> Tensor:                                                              │
│ >  460 │   │   return self._conv_forward(input, self.weight, self.bias)                                             │
│    461                                                                                                              │
│                                                                                                                     │
│ H:\SDNext_Master\venv\lib\site-packages\torch\nn\modules\conv.py:456 in _conv_forward                               │
│                                                                                                                     │
│    455 │   │   │   │   │   │   │   _pair(0), self.dilation, self.groups)                                            │
│ >  456 │   │   return F.conv2d(input, weight, bias, self.stride,                                                    │
│    457 │   │   │   │   │   │   self.padding, self.dilation, self.groups)                                            │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when
checking argument for argument weight in method wrapper_CUDA___slow_conv2d_forward)
MrMyth commented 11 months ago

Good day! i try to make animated image. But i got this. How to fix it? Thx. image

vladmandic commented 11 months ago

Good day! i try to make animated image. But i got this. How to fix it? Thx. image

screenshot with no description and not result in anything.

scpedicini commented 11 months ago

For the record, I could never get animatediff to work on any other sampler besides DDIM on most SD 1.5 models I tried with the mm_sd_v15_v2.ckpt motion module in SD.Next. It would generate a completely unrelated grid of images unless I swapped back to DDIM.

Extension creator recommends Pad prompt/negative prompt to be same length be set under optimizations but this flag has been removed from UI settings in SD.Next so I ended up dual-booting with auto1111 which doesn't seem to have these issues.

vladmandic commented 11 months ago

For the record, I could never get animatediff to work on any other sampler besides DDIM on most SD 1.5 models I tried with the mm_sd_v15_v2.ckpt motion module in SD.Next. It would generate a completely unrelated grid of images unless I swapped back to DDIM.

Extension creator recommends Pad prompt/negative prompt to be same length be set under optimizations but this flag has been removed from UI settings in SD.Next so I ended up dual-booting with auto1111 which doesn't seem to have these issues.

That is not helpful at all, especially since you never reported those issues and instead just comment on unrelated thread.

scpedicini commented 11 months ago

@vladmandic Three things

  1. It's not necessarily unrelated, his image of grids clearly are images with different seeds.

  2. Also I mentioned Pad prompt/negative prompt to be same length as a possible avenue of exploration.

  3. I didn't raise the issue because I'd read an earlier thread where you expressed frustration with the AnimateDiff/AnimateDiff extension changing so much that it was difficult for you to get working properly - so I didn't want to add to your workload.

If you would like, I can put in a new issue for the issues I was encountering.

EDIT: Ah, I see what you're saying - I assumed this thread was created by @MrMyth

vladmandic commented 11 months ago

Pad prompt has some impact to performance, none to functionality. So thats definitely not related.

scpedicini commented 11 months ago

@vladmandic

From Animatediff Extension github: https://github.com/continue-revolution/sd-webui-animatediff#how-to-use

"Enable Pad prompt/negative prompt to be same length in Settings/Optimization and click Apply settings. You must do this to prevent generating two separate unrelated GIFs."

That sounds like a functionality difference not a performance one - at least from the extension's author's perspective.

vladmandic commented 11 months ago

Extension does not do prompt parsing, application does. Besides, easy to test - use short prompt, so both positive and negative are below 77 tokens.

vladmandic commented 10 months ago

some improvements have been made to schedulers for animatediff in dev branch, will be merged to master in the next release.