[Bug]: Controlnet passed incorrect path. Video frames directory appears to be repeated.

[lemontheme]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

Have you read FAQ on README?

• [X] I have updated WebUI and this extension to the latest version

What happened?

I'm trying to get v2v to work.

Frame extraction is working as expected. Frames are saved to <webui_directory>/GIF/<video name>-<hash>/*.jpg.

Controlnet complains it can't read the frame images. That's because the image paths do not exist. If you look closely, they contain a repeat of the frames batch path.

I've provided the traceback below.

I did some poking around with the debugger, but since I'm new to the code base, I can't figure out where Controlnet is building the image path or where animatediff is passing it in.

Apologies in advance if this is actually a forge issue instead.

Steps to reproduce the problem

1. Upload an mpv to animatediff
2. Enable animatediff
3. Select SD checkpoint
4. Select animatediff model
5. Enable controlnet unit.
6. Do not upload reference image to controlnet unit.
7. Generate

What should have happened?

Controlnet uses the correct image path.

Commit where the problem happens

webui: 29be1da7 extension: b20f751

What browsers do you use to access the UI ?

No response

Command Line Arguments

No

Console logs

2024-03-21 17:24:39,683 - ControlNet - INFO - ControlNet Input Mode: InputMode.BATCH
2024-03-21 17:24:39,684 - ControlNet - INFO - Try to read image: GIF/pexels-cup-of-coffee-7657138 1080p-dca7761e/GIF/pexels-cup-of-coffee-7657138 1080p-dca7761e/000000001.jpg
[ WARN:0@175.916] global loadsave.cpp:248 findDecoder imread_('GIF/pexels-cup-of-coddee-7657138 1080p-dca7761e/GIF/pexels-cup-of-coffee-7657138 1080p-dca7761e/000000001.jpg'): can't open/read file: check file path/integrity
*** Error running process: /home/adriaan/Code/cloned/use/stable-diffusion/automatic-webui/extensions-builtin/sd_forge_controlnet/scripts/controlnet.py
    Traceback (most recent call last):
      File "/home/adriaan/Code/cloned/use/stable-diffusion/automatic-webui/modules/scripts.py", line 803, in process
        script.process(p, *script_args)
      File "/home/adriaan/Code/cloned/use/stable-diffusion/automatic-webui/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "/home/adriaan/Code/cloned/use/stable-diffusion/automatic-webui/extensions-builtin/sd_forge_controlnet/scripts/controlnet.py", line 548, in process
        self.process_unit_after_click_generate(p, unit, params, *args, **kwargs)
      File "/home/adriaan/Code/cloned/use/stable-diffusion/automatic-webui/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "/home/adriaan/Code/cloned/use/stable-diffusion/automatic-webui/extensions-builtin/sd_forge_controlnet/scripts/controlnet.py", line 298, in process_unit_after_click_generate
        input_list, resize_mode = self.get_input_data(p, unit, preprocessor, h, w)
      File "/home/adriaan/Code/cloned/use/stable-diffusion/automatic-webui/extensions-builtin/sd_forge_controlnet/scripts/controlnet.py", line 182, in get_input_data
        img = np.ascontiguousarray(cv2.imread(img_path)[:, :, ::-1]).copy()
    TypeError: 'NoneType' object is not subscriptable

Additional information

No response

[lemontheme]

In _animatediffui.py, the batch_image_dir path is set correctly.

        cn_units = get_controlnet_units(p)
        min_batch_in_cn = -1
        for cn_unit in cn_units:
            # batch path broadcast
            if (cn_unit.input_mode.name == 'SIMPLE' and cn_unit.image is None) or \
               (cn_unit.input_mode.name == 'BATCH' and not cn_unit.batch_image_dir) or \
               (cn_unit.input_mode.name == 'MERGE' and not cn_unit.batch_input_gallery):
                if not self.video_path:
                    extract_frames_from_video(self)
                cn_unit.input_mode = cn_unit.input_mode.__class__.BATCH
                cn_unit.batch_image_dir = self.video_path
                breakpoint()

pdb output:

(Pdb) cn_unit.batch_image_dir
'GIF/pexels-cup-of-couple-7657138 1080p-6b899dcb'

[continue-revolution]

Remove white space in your video filename , use the path to extracted frames instead for now. In settings/AnimateDiff, make the extraction path to be blank instead of “GIF” (“GIF” was because I made some stupid typo before)

[lemontheme]

Hi there! Just found the culprit. It's this line in forge's controlnet.py: batch_modifer() (three or so lines above) is returning the full path instead of the file's basename. Next, that path gets needlessly joined with unit.batch_image_dir.

[lemontheme]

@continue-revolution That seems to get me past this step, yay! (I'm facing CUDA side-asserts now, but I doubt it's related.)

Just to be sure, what do you mean by this?

use the path to extracted frames instead for now.

I'm not sure where I'm supposed to specify that. Thanks!