Out of Memory Errors / "AttributeError: 'AnimateDiffProcess' object has no attribute 'text_cond'" or "'multi_cond'" when running with ControlNet V2V #296
I am trying to run AnimateDiff with ControlNet V2V. Both ControlNet and AnimateDiff work fine separately. I am following these instructions almost exactly, save for making the prompt slightly more SFW (scroll down to "Video to Video Using ControlNet’s ‘Video Source’ Upload"). In summary, the guide provides an example MP4 file that is a 113 frame/30 fps/1MB video with very little movement, and has the user load that into ControlNet with the OpenPose model.
As background, I'm running A1111 on Windows 11 with an RTX 4060 Ti 16GB.
A1111 v1.8.0
python: 3.10.13
torch: 2.2.1+cu118
xformers: 0.0.24+cu118
gradio: 3.41.2
My image previews are off, and my prompts are padded.
When I try generating a GIF using the guide's suggested resolution of 768x768, I get the below output, ending with an out of memory error. OK, maybe 768x768 is too much for a 113 frame GIF even with 16GB VRAM, but I did notice an 'ffmpeg' error, a weird "ValueError: controlnet is enabled but no input image is given" error (I didn't think I was supposed to drop anything into the Controlnet interface for V2V), and a "AttributeError: 'NoneType' object has no attribute 'multi_cond'" error.
---
2024-03-04 12:02:53,491 - AnimateDiff - INFO - AnimateDiff process start.
2024-03-04 12:02:53,491 - AnimateDiff - INFO - Motion module already injected. Trying to restore.
2024-03-04 12:02:53,492 - AnimateDiff - INFO - Restoring DDIM alpha.
2024-03-04 12:02:53,492 - AnimateDiff - INFO - Removing motion module from SD1.5 UNet input blocks.
2024-03-04 12:02:53,492 - AnimateDiff - INFO - Removing motion module from SD1.5 UNet output blocks.
2024-03-04 12:02:53,493 - AnimateDiff - INFO - Removing motion module from SD1.5 UNet middle block.
2024-03-04 12:02:53,493 - AnimateDiff - INFO - Removal finished.
2024-03-04 12:02:53,502 - AnimateDiff - INFO - Injecting motion module mm_sd15_v2.safetensors into SD1.5 UNet middle block.
2024-03-04 12:02:53,502 - AnimateDiff - INFO - Injecting motion module mm_sd15_v2.safetensors into SD1.5 UNet input blocks.
2024-03-04 12:02:53,503 - AnimateDiff - INFO - Injecting motion module mm_sd15_v2.safetensors into SD1.5 UNet output blocks.
2024-03-04 12:02:53,503 - AnimateDiff - INFO - Setting DDIM alpha.
2024-03-04 12:02:53,506 - AnimateDiff - INFO - Injection finished.
2024-03-04 12:02:53,507 - AnimateDiff - ERROR - [AnimateDiff] Error extracting frames via ffmpeg: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'GIFC:\\Users\\ernie\\AppData\\Local\\Temp\\gradio\\185e036dbc445c446fae79292baaa6316036fd21\\gil512.mp4-82824f3b', fall back to OpenCV.
2024-03-04 12:02:53,507 - AnimateDiff - INFO - Attempting to extract frames via OpenCV from C:\Users\ernie\AppData\Local\Temp\gradio\185e036dbc445c446fae79292baaa6316036fd21\gil512.mp4 to GIFC:\Users\ernie\AppData\Local\Temp\gradio\185e036dbc445c446fae79292baaa6316036fd21\gil512.mp4-82824f3b
*** Error running before_process: C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
Traceback (most recent call last):
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 102, in extract_frames_from_video
ffmpeg_extract_frames(params.video_source, params.video_path)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 73, in ffmpeg_extract_frames
tmp_frame_dir.mkdir(parents=True, exist_ok=True)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\pathlib.py", line 1175, in mkdir
self._accessor.mkdir(self, mode)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'GIFC:\\Users\\ernie\\AppData\\Local\\Temp\\gradio\\185e036dbc445c446fae79292baaa6316036fd21\\gil512.mp4-82824f3b'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\ernie\stable-diffusion-webui\modules\scripts.py", line 776, in before_process
script.before_process(p, *script_args)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 64, in before_process
params.set_p(p)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_ui.py", line 169, in set_p
extract_frames_from_video(self)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 105, in extract_frames_from_video
cv2_extract_frames(params.video_source, params.video_path)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 84, in cv2_extract_frames
tmp_frame_dir.mkdir(parents=True, exist_ok=True)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\pathlib.py", line 1175, in mkdir
self._accessor.mkdir(self, mode)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'GIFC:\\Users\\ernie\\AppData\\Local\\Temp\\gradio\\185e036dbc445c446fae79292baaa6316036fd21\\gil512.mp4-82824f3b'
---
2024-03-04 12:02:53,537 - ControlNet - INFO - unit_separate = False, style_align = False
2024-03-04 12:02:53,537 - ControlNet - INFO - Loading model from cache: control_v11p_sd15_openpose [cab727d4]
*** Error running process: C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\Users\ernie\stable-diffusion-webui\modules\scripts.py", line 784, in process
script.process(p, *script_args)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1275, in process
self.controlnet_hack(p)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1260, in controlnet_hack
self.controlnet_main_entry(p)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 949, in controlnet_main_entry
input_image, resize_mode = Script.choose_input_image(p, unit, idx)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 726, in choose_input_image
raise ValueError("controlnet is enabled but no input image is given")
ValueError: controlnet is enabled but no input image is given
---
0%| | 0/20 [00:00<?, ?it/s]*** Error executing callback cfg_denoiser_callback for C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
Traceback (most recent call last):
File "C:\Users\ernie\stable-diffusion-webui\modules\script_callbacks.py", line 230, in cfg_denoiser_callback
c.callback(params)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_infv2v.py", line 90, in animatediff_on_cfg_denoiser
ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
AttributeError: 'NoneType' object has no attribute 'multi_cond'
---
0%| | 0/20 [00:03<?, ?it/s]
*** Error completing request
*** Arguments: ('task(x7yeo6ztcmh8ojz)', <gradio.routes.Request object at 0x000002AC4C9A5B70>, '(best quality, masterpiece, 1girl, t-shrt, jeans, highest detailed)full body photo, ultra detailed, (textured_clothing), black_background, (intricate details, hyperdetailed:1.15), detailed, (official art, extreme detailed, highest detailed), ', 'EasyNegative, bad-hands-5 ', [], 20, 'DPM++ 2M Karras', 1, 1, 6, 768, 768, False, 0.7, 2, 'Latent', 0, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], 0, False, '', 0.8, 862816124, False, -1, 0, 0, 0, False, 'CodeFormer', False, False, {'ad_model': 'face_yolov8n.pt', 'ad_model_classes': '', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M Karras', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, {'ad_model': 'None', 'ad_model_classes': '', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M Karras', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, <scripts.animatediff_ui.AnimateDiffProcess object at 0x000002AC4C9A4520>, UiControlNetUnit(enabled=True, module='openpose_full', model='control_v11p_sd15_openpose [cab727d4]', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=512, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=True, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), None, '', None, True, False, False, False, False, False, 0, 0, '0', 0, False, True, 0, 'Portrait of a [gender]', 'blurry', 20, ['DPM++ 2M Karras'], '', 0, None, 1, 1, '', False, False, False, 1, 0, 'Portrait of a [gender]', 'blurry', 20, ['DPM++ 2M Karras'], '', 0, None, '', None, True, False, False, False, False, False, 0, 0, '0', 0, False, True, 0, 'Portrait of a [gender]', 'blurry', 20, ['DPM++ 2M Karras'], '', 0, None, 1, 1, '', False, False, False, 1, 0, 'Portrait of a [gender]', 'blurry', 20, ['DPM++ 2M Karras'], '', 0, None, '', None, True, False, False, False, False, False, 0, 0, '0', 0, False, True, 0, 'Portrait of a [gender]', 'blurry', 20, ['DPM++ 2M Karras'], '', 0, None, 1, 1, '', False, False, False, 1, 0, 'Portrait of a [gender]', 'blurry', 20, ['DPM++ 2M Karras'], '', 0, None, 1, 1, '', 1, 1, ['After Upscaling/Before Restore Face'], 0, 'Portrait of a [gender]', 'blurry', 20, ['DPM++ 2M Karras'], '', 0, None, False, '0', 'C:\\Users\\ernie\\stable-diffusion-webui\\models\\roop\\inswapper_128.onnx', 'CodeFormer', 1, '', 1, 1, False, True, False, False, 'positive', 'comma', 0, False, False, 'start', '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, False, False, False, 0, False, None, None, False, None, None, False, None, None, False, 50) {}
Traceback (most recent call last):
File "C:\Users\ernie\stable-diffusion-webui\modules\call_queue.py", line 57, in f
res = list(func(*args, **kwargs))
File "C:\Users\ernie\stable-diffusion-webui\modules\call_queue.py", line 36, in f
res = func(*args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\modules\txt2img.py", line 110, in txt2img
processed = processing.process_images(p)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-prompt-history\lib_history\image_process_hijacker.py", line 15, in process_images
return original_function(p)
File "C:\Users\ernie\stable-diffusion-webui\modules\processing.py", line 785, in process_images
res = process_images_inner(p)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 48, in processing_process_images_hijack
return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\modules\processing.py", line 921, in process_images_inner
samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
File "C:\Users\ernie\stable-diffusion-webui\modules\processing.py", line 1257, in sample
samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
File "C:\Users\ernie\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 234, in sample
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "C:\Users\ernie\stable-diffusion-webui\modules\sd_samplers_common.py", line 261, in launch_sampling
return func()
File "C:\Users\ernie\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 234, in <lambda>
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\modules\sd_samplers_cfg_denoiser.py", line 237, in forward
x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict(cond_in, image_cond_in))
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\modules\sd_hijack_utils.py", line 18, in <lambda>
setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
File "C:\Users\ernie\stable-diffusion-webui\modules\sd_hijack_utils.py", line 32, in __call__
return self.__orig_func(*args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
x_recon = self.model(x_noisy, t, **cond)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
out = self.diffusion_model(x, t, context=cc)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\modules\sd_unet.py", line 91, in UNetModel_forward
return original_forward(self, x, timesteps, context, *args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 797, in forward
h = module(h, emb, context)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 84, in forward
x = layer(x, context)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 334, in forward
x = block(x, context=context[i])
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 269, in forward
return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
File "C:\Users\ernie\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 121, in checkpoint
return CheckpointFunction.apply(func, len(inputs), *args)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\autograd\function.py", line 553, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "C:\Users\ernie\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 136, in forward
output_tensors = ctx.run_function(*ctx.input_tensors)
File "C:\Users\ernie\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 272, in _forward
x = self.attn1(self.norm1(x), context=context if self.disable_self_attn else None) + x
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\modules\sd_hijack_optimizations.py", line 501, in xformers_attention_forward
return self.to_out(out)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\container.py", line 217, in forward
input = module(input)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\extensions-builtin\Lora\networks.py", line 500, in network_Linear_forward
return originals.Linear_forward(self, input)
File "C:\Users\ernie\anaconda3\envs\autosd5\lib\site-packages\torch\nn\modules\linear.py", line 116, in forward
return F.linear(input, self.weight, self.bias)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.24 GiB. GPU 0 has a total capacity of 16.00 GiB of which 0 bytes is free. Of the allocated memory 14.52 GiB is allocated by PyTorch, and 210.67 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
---
When I try reducing the resolution to 320x320, I get the same controlnet and AttributeErrors I mentioned above, but it then ends with " Shape mismatch, can't divide axis of length 226 in chunks of 16" (recall that this is a 113 frame video).
Finally, when I try reducing the frames to 112, it runs, but after every step I get the below error message before it continues. And then it ends at 100% with the bottom error message. It generates the frames but does not combine them into a GIF (again, on its own AnimateDiff does this fine), and the frames do not appear to be adhering to the poses in the video.
5%|██████████▊ | 1/20 [00:07<02:13, 7.03s/it]*** Error executing callback cfg_denoiser_callback for C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py | 0/20 [00:00<?, ?it/s]
Traceback (most recent call last):
File "C:\Users\ernie\stable-diffusion-webui\modules\script_callbacks.py", line 230, in cfg_denoiser_callback
c.callback(params)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_infv2v.py", line 163, in animatediff_on_cfg_denoiser
cfg_params.text_cond = ad_params.text_cond
AttributeError: 'AnimateDiffProcess' object has no attribute 'text_cond'
---
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [01:47<00:00, 5.37s/it]
*** Error running postprocess_batch_list: C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py██████████████████████████████████████████████████████████████████████████████████████| 20/20 [01:40<00:00, 5.19s/it]
Traceback (most recent call last):
File "C:\Users\ernie\stable-diffusion-webui\modules\scripts.py", line 832, in postprocess_batch_list
script.postprocess_batch_list(p, pp, *script_args, **kwargs)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 80, in postprocess_batch_list
params.prompt_scheduler.save_infotext_img(p)
AttributeError: 'NoneType' object has no attribute 'save_infotext_img'
---
*** Error running postprocess: C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
Traceback (most recent call last):
File "C:\Users\ernie\stable-diffusion-webui\modules\scripts.py", line 816, in postprocess
script.postprocess(p, processed, *script_args)
File "C:\Users\ernie\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 90, in postprocess
params.prompt_scheduler.save_infotext_txt(res)
AttributeError: 'NoneType' object has no attribute 'save_infotext_txt'
I am trying to run AnimateDiff with ControlNet V2V. Both ControlNet and AnimateDiff work fine separately. I am following these instructions almost exactly, save for making the prompt slightly more SFW (scroll down to "Video to Video Using ControlNet’s ‘Video Source’ Upload"). In summary, the guide provides an example MP4 file that is a 113 frame/30 fps/1MB video with very little movement, and has the user load that into ControlNet with the OpenPose model.
As background, I'm running A1111 on Windows 11 with an RTX 4060 Ti 16GB. A1111 v1.8.0 python: 3.10.13 torch: 2.2.1+cu118 xformers: 0.0.24+cu118 gradio: 3.41.2
My image previews are off, and my prompts are padded.
When I try generating a GIF using the guide's suggested resolution of 768x768, I get the below output, ending with an out of memory error. OK, maybe 768x768 is too much for a 113 frame GIF even with 16GB VRAM, but I did notice an 'ffmpeg' error, a weird "ValueError: controlnet is enabled but no input image is given" error (I didn't think I was supposed to drop anything into the Controlnet interface for V2V), and a "AttributeError: 'NoneType' object has no attribute 'multi_cond'" error.
When I try reducing the resolution to 320x320, I get the same controlnet and AttributeErrors I mentioned above, but it then ends with " Shape mismatch, can't divide axis of length 226 in chunks of 16" (recall that this is a 113 frame video).
Finally, when I try reducing the frames to 112, it runs, but after every step I get the below error message before it continues. And then it ends at 100% with the bottom error message. It generates the frames but does not combine them into a GIF (again, on its own AnimateDiff does this fine), and the frames do not appear to be adhering to the poses in the video.