[Bug]: Controlnet doesn't work with Animatediff in Forge

[afrofail]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

Have you read FAQ on README?

• [X] I have updated WebUI and this extension to the latest version

What happened?

The built in forge Controlnet has errors and wont use video source information within animatediff. I tested in Automatic1111 with the Automatic1111 Animatediff extension and can load a video source, have controlnet preprocess the frames and generate in Txt2Img. Another issue is "sd15_t2v_beta" motion model also works in Automatic1111 but does not work at all in Forge.

Steps to reproduce the problem

1. txt2img
2. enable animatediff using "v3_sd15_mm.ckpt"
3. enable controlnet depth or tile
4. use video source
5. error

What should have happened?

In Automatic1111, controlnet works perfect along with using a video source and input. This should the built in controlnet in forge should have preprocessed each frame (24) from the video source in animatediff and generated with controlnet

Commit where the problem happens

webui: Forge extension: AnimateDiff

What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome

Command Line Arguments

no

Console logs

*** Error running before_process: M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\scripts.py", line 795, in before_process
        script.before_process(p, *script_args)
      File "M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py", line 57, in before_process
        params.set_p(p)
      File "M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_ui.py", line 176, in set_p
        cn_unit_batch_num = len(shared.listfiles(cn_unit.batch_image_dir))
      File "M:\stable-diffusion-webui-forge\modules\util.py", line 13, in listfiles
        filenames = [os.path.join(dirname, x) for x in sorted(os.listdir(dirname), key=natural_sort_key) if not x.startswith(".")]
    NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\Christiani Peluso\\Desktop\\Burning Palace Test\\Test 4\\Comp.mp4'

---
2024-02-24 02:23:31,566 - ControlNet - INFO - ControlNet Input Mode: InputMode.BATCH
*** Error running process: M:\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\scripts.py", line 803, in process
        script.process(p, *script_args)
      File "M:\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "M:\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 548, in process
        self.process_unit_after_click_generate(p, unit, params, *args, **kwargs)
      File "M:\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "M:\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 298, in process_unit_after_click_generate
        input_list, resize_mode = self.get_input_data(p, unit, preprocessor, h, w)
      File "M:\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 175, in get_input_data
        batch_image_files = shared.listfiles(unit.batch_image_dir)
      File "M:\stable-diffusion-webui-forge\modules\util.py", line 13, in listfiles
        filenames = [os.path.join(dirname, x) for x in sorted(os.listdir(dirname), key=natural_sort_key) if not x.startswith(".")]
    NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\Christiani Peluso\\Desktop\\Burning Palace Test\\Test 4\\Comp.mp4'

---
2024-02-24 02:23:31,568 - AnimateDiff - INFO - Setting DDIM alpha.
*** Error running process_before_every_sampling: M:\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\scripts.py", line 835, in process_before_every_sampling
        script.process_before_every_sampling(p, *script_args, **kwargs)
      File "M:\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "M:\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 555, in process_before_every_sampling
        self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
    KeyError: 0

---
2024-02-24 02:23:31,571 - AnimateDiff - INFO - Injecting motion module v3_sd15_mm.ckpt into SD1.5 UNet.
To load target model BaseModel
To load target model MotionWrapper
Begin to load 2 models
Reuse 1 loaded models
Reuse 1 loaded models
[Memory Management] Current Free Memory (MB) =  9887.630859375
[Memory Management] Model Memory (MB) =  1634.609375
[Memory Management] Estimated Inference Memory (MB) =  6040.655268554688
[Memory Management] Estimated Remaining Memory (MB) =  2212.3662158203124
[Memory Management] Current Free Memory (MB) =  8241.685546875
[Memory Management] Model Memory (MB) =  0.0
[Memory Management] Estimated Inference Memory (MB) =  6040.655268554688
[Memory Management] Estimated Remaining Memory (MB) =  2201.0302783203124
Moving model(s) has taken 0.95 seconds
  0%|                                                                                            | 0/8 [00:00<?, ?it/s]*** Error executing callback cfg_denoiser_callback for M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\script_callbacks.py", line 233, in cfg_denoiser_callback
        c.callback(params)
      File "M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_infv2v.py", line 85, in animatediff_on_cfg_denoiser
        ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
    AttributeError: 'NoneType' object has no attribute 'multi_cond'

---
 12%|██████████▌                                                                         | 1/8 [00:05<00:39,  5.67s/it]*** Error executing callback cfg_denoiser_callback for M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\script_callbacks.py", line 233, in cfg_denoiser_callback
        c.callback(params)
      File "M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_infv2v.py", line 85, in animatediff_on_cfg_denoiser
        ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
    AttributeError: 'NoneType' object has no attribute 'multi_cond'

---
 25%|█████████████████████                                                               | 2/8 [00:11<00:34,  5.76s/it]*** Error executing callback cfg_denoiser_callback for M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\script_callbacks.py", line 233, in cfg_denoiser_callback
        c.callback(params)
      File "M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_infv2v.py", line 85, in animatediff_on_cfg_denoiser
        ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
    AttributeError: 'NoneType' object has no attribute 'multi_cond'

---
 38%|███████████████████████████████▌                                                    | 3/8 [00:17<00:28,  5.65s/it]*** Error executing callback cfg_denoiser_callback for M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\script_callbacks.py", line 233, in cfg_denoiser_callback
        c.callback(params)
      File "M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_infv2v.py", line 85, in animatediff_on_cfg_denoiser
        ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
    AttributeError: 'NoneType' object has no attribute 'multi_cond'

---
 50%|██████████████████████████████████████████                                          | 4/8 [00:22<00:22,  5.63s/it]*** Error executing callback cfg_denoiser_callback for M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\script_callbacks.py", line 233, in cfg_denoiser_callback
        c.callback(params)
      File "M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_infv2v.py", line 85, in animatediff_on_cfg_denoiser
        ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
    AttributeError: 'NoneType' object has no attribute 'multi_cond'

---
 62%|████████████████████████████████████████████████████▌                               | 5/8 [00:28<00:16,  5.56s/it]*** Error executing callback cfg_denoiser_callback for M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\script_callbacks.py", line 233, in cfg_denoiser_callback
        c.callback(params)
      File "M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_infv2v.py", line 85, in animatediff_on_cfg_denoiser
        ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
    AttributeError: 'NoneType' object has no attribute 'multi_cond'

---
 75%|███████████████████████████████████████████████████████████████                     | 6/8 [00:33<00:11,  5.63s/it]*** Error executing callback cfg_denoiser_callback for M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\script_callbacks.py", line 233, in cfg_denoiser_callback
        c.callback(params)
      File "M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_infv2v.py", line 85, in animatediff_on_cfg_denoiser
        ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
    AttributeError: 'NoneType' object has no attribute 'multi_cond'

---
 88%|█████████████████████████████████████████████████████████████████████████▌          | 7/8 [00:39<00:05,  5.53s/it]*** Error executing callback cfg_denoiser_callback for M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\script_callbacks.py", line 233, in cfg_denoiser_callback
        c.callback(params)
      File "M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_infv2v.py", line 85, in animatediff_on_cfg_denoiser
        ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
    AttributeError: 'NoneType' object has no attribute 'multi_cond'

---
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:44<00:00,  5.60s/it]
*** Error running postprocess_batch_list: M:\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\scripts.py", line 859, in postprocess_batch_list
        script.postprocess_batch_list(p, pp, *script_args, **kwargs)
      File "M:\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "M:\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 561, in postprocess_batch_list
        self.process_unit_after_every_sampling(p, unit, self.current_params[i], pp, *args, **kwargs)
    KeyError: 0

---
*** Error running postprocess_batch_list: M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\scripts.py", line 859, in postprocess_batch_list
        script.postprocess_batch_list(p, pp, *script_args, **kwargs)
      File "M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py", line 79, in postprocess_batch_list
        params.prompt_scheduler.save_infotext_img(p)
    AttributeError: 'NoneType' object has no attribute 'save_infotext_img'

---
*** Error running postprocess: M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "M:\stable-diffusion-webui-forge\modules\scripts.py", line 843, in postprocess
        script.postprocess(p, processed, *script_args)
      File "M:\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py", line 84, in postprocess
        params.prompt_scheduler.save_infotext_txt(res)
    AttributeError: 'NoneType' object has no attribute 'save_infotext_txt'

Additional information

No response

[continue-revolution]

there is some bug in it if you directly upload a video. I should have fixed it in forge/dev branch in sd-webui-animatediff repository but I am not sure if it's working cuz I haven't had a chance to test it. Anyway, I will try to push a fix tomorrow.

[threeforces]

Same issue here...

[lemontheme]

I'm afraid the problem persists on dev. I'm on the latest commit of forge.

Let me know if there are specific parts of the traceback that would be helpful to share.

[PixelClassicist]

Same issue still persists here. Using Animatediff without controlnet works fine, but enabling Controlnet creates the following error:

\webui_forge_cu121_torch21\webui\extensions\sd-forge-animatediff\scripts\animatediff_infv2v.py", line 85, in animatediff_on_cfg_denoiser ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop) AttributeError: 'NoneType' object has no attribute 'multi_cond'

Would love to use this with Forge, so appreciate all the support on this.

[Rokkonrol]

It was working fine for about about two weeks after the fix, but for the past week or so I've been getting the same AttributeError: 'NoneType' object has no attribute 'multi_cond' error.

[continue-revolution]

I believe that this error has been fixed. please retry. if not, feel free to re-open this issue or open another issue.

[jameslh]

I had to modify line 131 in animatediff_infv2v.py because I was getting a cannot iterate NoneType error on that line.

from: if k == "control": to: if k == "control" and v is not None:

Edit: The above did not completely fix the issue and I am getting this error:

** Error executing callback cfg_denoiser_callback for J:\dev\python\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "J:\dev\python\stable-diffusion-webui-forge\modules\script_callbacks.py", line 233, in cfg_denoiser_callback
        c.callback(params)
      File "J:\dev\python\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_infv2v.py", line 100, in animatediff_on_cfg_denoiser
        cfg_params.text_cond, cfg_params.text_uncond = pad_cond_uncond(ad_params.text_cond, cfg_params.text_uncond)
    AttributeError: 'AnimateDiffProcess' object has no attribute 'text_cond'