[Feature]: AnimateDiff can't generate ControlNet Batch images properly

[Sivanx1]

Expected behavior

Two weeks ago, I was generating turntable characters with A1111/AnimateDiff very well but yesterday after updating the extension, AnimateDiff has started to generate totally different results and there's no way to do the same workflow anymore. I was using AnimateDiff with a series of the batch images with the ControlNet tabs. The first CN tab for NormalMap, the second tab for Canny, the third tab for OpenPose sequences and the fourth tab for a single image as reference. That was work very well to generate a great turntable animation with AnimateDiff but after my recent update yesterday, the same workflow with the same prompts and batch sequences gives me entirely different results which looks totally different than I've expected. I use Maya render to produce NormalMap sequences of a 3d model, I use ControlNet to generate the Canny and OpenPose sequences, Here is one of my past result: https://youtu.be/WwNwwnGeF_A

This workflow not to work with AnimateDiff anymore. AnimateDiff was generating a stable result with the outfits and the rest of the parts of the character. Would you tell me what happened the results are no more similar anymore? AnimateDiff generating the prompt only despite using 'ControlNet is more Important'.

[continue-revolution]

What have you updated? You can always revert back. Don’t update ControlNet because it’s not compatible now and I’m working on an aggressive update.

1. Take a look at your previous infotext and make sure that you are still using the same A1111 version
2. In extensions/sd-webui-animatediff, run git checkout v1.13.0 (or any other version tag) on your terminal to revert back AnimateDiff. Don’t worry you can still update.
3. In extensions/sd-webui-controlnet, you need the commit hash in https://github.com/continue-revolution/sd-webui-animatediff/issues/363#issuecomment-1845671560

Let me know if the problem still exists.

[pitkevich80]

Same problem, AnimateDiff doesn't work with controlnet anymore. ** Error running process: J:\Stable Diffusion\webui\extensions\sd-webui-controlnet\scripts\controlnet.py Traceback (most recent call last): File "J:\Stable Diffusion\webui\modules\scripts.py", line 718, in process script.process(p, script_args) File "J:\Stable Diffusion\webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1117, in process self.controlnet_hack(p) File "J:\Stable Diffusion\webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1102, in controlnet_hack self.controlnet_main_entry(p) File "J:\Stable Diffusion\webui\extensions\sd-webui-animatediff\scripts\animatediff_cn.py", line 248, in hacked_main_entry model_net.reset() AttributeError: 'ControlModel' object has no attribute 'reset'

What version of Controlnet work with Animatediff for now? (ANd how can i install old version of Controlnet) ?

[Rokkonrol]

Changing the commit hash for ControlNet worked for me. I used the hash mentioned by @continue-revolution. I didn't change anything for AnimateDiff.

1. Navigate to extensions/sd-webui-controlnet in terminal.
2. git checkout -b new_branch 10bd9b25f62deab9acb256301bbf3363c42645e7
3. git pull

For reference, I'm using the regular v1.7.0 master branch of AUTOMATIC1111.

[continue-revolution]

For the time being, please checkout old commits for CN. You can always go back easily. I don't have a lot of availability recently so please temporariily apply this workaround.

[pitkevich80]

Yeah, roll back to old version CN.. Work for now. Please update as long as NEW CN will work with ADiff. Thank you.

[fulleclip]

After several Controlnet updates in recent days and today from Animatediff, the error continues to appear, any suggestions?

2024-02-17 21:40:18,434 - ControlNet - INFO - ControlNet model control_v11p_sd15_openpose cab727d4 loaded. ** Error running process: D:\sd1\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py Traceback (most recent call last): File "D:\sd1\stable-diffusion-webui\modules\scripts.py", line 718, in process script.process(p, script_args) File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1143, in process self.controlnet_hack(p) File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1128, in controlnet_hack self.controlnet_main_entry(p) File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_cn.py", line 248, in hacked_main_entry model_net.reset() AttributeError: 'ControlModel' object has no attribute 'reset'

[eliasprompt]

After several Controlnet updates in recent days and today from Animatediff, the error continues to appear, any suggestions?

2024-02-17 21:40:18,434 - ControlNet - INFO - ControlNet model control_v11p_sd15_openpose cab727d4 loaded. ** Error running process: D:\sd1\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py Traceback (most recent call last): File "D:\sd1\stable-diffusion-webui\modules\scripts.py", line 718, in process script.process(p, script_args) File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1143, in process self.controlnet_hack(p) File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1128, in controlnet_hack self.controlnet_main_entry(p) File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_cn.py", line 248, in hacked_main_entry model_net.reset() AttributeError: 'ControlModel' object has no attribute 'reset'

Same issue here, I downgraded CN to older version like some people wrote before but didn't worked. Hope new update of AnimateDiff fix this issue with Controlnet

[fulleclip]

New update of both extensions and when it seemed that everything was going well, this appears

2024-02-19 21:32:50,823 - ControlNet - INFO - ControlNet Hooked - Time = 43.25327801704407 0%| | 0/25 [00:01<?, ?it/s] Error completing request Arguments: ('task(z5dth55jksegmp2)', 'dog jump', '', [], 25, 'DPM++ 2M Karras', 1, 1, 13, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], <gradio.routes.Request object at 0x000001AD20AF5B10>, 0, False, '', 0.8, -1, False, -1, 0, 0, 0, False, False, False, {'ad_model': 'face_yolov8n.pt', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M Karras', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, {'ad_model': 'None', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M Karras', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, <scripts.animatediff_ui.AnimateDiffProcess object at 0x000001AD20AF78E0>, UiControlNetUnit(enabled=True, module='openpose_full', model='control_v11p_sd15_openpose [cab727d4]', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=512, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='ControlNet is more important', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), False, False, 'positive', 'comma', 0, False, False, 'start', '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False, None, None, False, None, None, False, None, None, False, 50) {} Traceback (most recent call last): File "D:\sd1\stable-diffusion-webui\modules\call_queue.py", line 57, in f res = list(func(*args, kwargs)) File "D:\sd1\stable-diffusion-webui\modules\call_queue.py", line 36, in f res = func(*args, *kwargs) File "D:\sd1\stable-diffusion-webui\modules\txt2img.py", line 55, in txt2img processed = processing.process_images(p) File "D:\sd1\stable-diffusion-webui\modules\processing.py", line 734, in process_images res = process_images_inner(p) File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_cn.py", line 119, in hacked_processing_process_images_hijack return getattr(processing, '__controlnet_original_process_images_inner')(p, args, kwargs) File "D:\sd1\stable-diffusion-webui\modules\processing.py", line 868, in process_images_inner samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts) File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 438, in process_sample return process.sample_before_CN_hack(*args, kwargs) File "D:\sd1\stable-diffusion-webui\modules\processing.py", line 1142, in sample samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x)) File "D:\sd1\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in sample samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, extra_params_kwargs)) File "D:\sd1\stable-diffusion-webui\modules\sd_samplers_common.py", line 261, in launch_sampling return func() File "D:\sd1\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, extra_params_kwargs)) File "D:\sd1\stable-diffusion-webui\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "D:\sd1\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m denoised = model(x, sigmas[i] s_in, extra_args) File "D:\sd1\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_infv2v.py", line 250, in mm_cfg_forward x_out = mm_sd_forward(self, x_in, sigma_in, cond_in, image_cond_in, make_condition_dict) # hook File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_infv2v.py", line 160, in mm_sd_forward out = self.inner_model( File "D:\sd1\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "D:\sd1\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward eps = self.get_eps(input c_in, self.sigma_to_t(sigma), kwargs) File "D:\sd1\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps return self.inner_model.apply_model(*args, kwargs) File "D:\sd1\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in setattr(resolved_obj, func_path[-1], lambda *args, *kwargs: self(args, kwargs)) File "D:\sd1\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in call return self.__orig_func(args, kwargs) File "D:\sd1\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model x_recon = self.model(x_noisy, t, cond) File "D:\sd1\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "D:\sd1\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward out = self.diffusion_model(x, t, context=cc) File "D:\sd1\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 862, in forward_webui raise e File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 859, in forward_webui return forward(args, kwargs) File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 581, in forward control = param.control_model( File "D:\sd1\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\cldm.py", line 31, in forward return self.control_model(*args, *kwargs) File "D:\sd1\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "D:\sd1\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\cldm.py", line 311, in forward h += guided_hint RuntimeError: The size of tensor a (32) must match the size of tensor b (20) at non-singleton dimension 0

---

[fulleclip]

After the last update of animatediff everything works correctly again, thanks, good job