continue-revolution / sd-webui-animatediff

AnimateDiff for AUTOMATIC1111 Stable Diffusion WebUI
Other
3.11k stars 258 forks source link

[Bug]: API调用插件v2v,controlnet is enabled but no input image is given #497

Open Chi-yuki opened 7 months ago

Chi-yuki commented 7 months ago

Is there an existing issue for this?

Have you read FAQ on README?

What happened?

我通过API调用插件,进行v2v,controlnet不能正常工作

Steps to reproduce the problem

  1. 启动sd webui
  2. 通过以下代码调用api

import requests import json

url = "http://127.0.0.1:7890/sdapi/v1/txt2img"

payload = { "alwayson_scripts": { "AnimateDiff": { "args": [ { "model": "mm_sd15_v3.safetensors", "format": ["PNG", "MP4"], "enable": True, "video_length": 1, "fps": 1, "closed_loop": "N", "batch_size": 1, "video_path": "D:/frame", "mask_path": "D:/mask" } ] }, "controlnet": { "args": [ { "enabled": True, "model": "control_v11p_sd15_inpaint [ebff9138]", "module": "inpaint_only", "pixel_perfect": True, "weight": 1 } ] } }, "batch_size": 8, "cfg_scale": 7, "denoising_strength": 0.7, "height": 1280, "prompt": "beautiful_lighting,1girl,", "sampler_name" : "Euler a", "seed" : -1, "steps" : 1, "width" : 720 }

response = requests.post(url, json=payload)

What should have happened?

正常通过controlnet控制生成动画

Commit where the problem happens

webui: 版本: v1.8.0  •  python: 3.10.6  •  torch: 2.1.2+cu121  •  xformers: 0.0.23.post1  •  gradio: 3.41.2  •  checkpoint: 86618977ac

extension: master | a81565d9

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

set COMMANDLINE_ARGS=--port 7890 ^
--listen ^
--enable-insecure-extension-access ^
--xformers ^
--api

Console logs

2024-04-10 11:22:44,162 - AnimateDiff - INFO - AnimateDiff process start.
2024-04-10 11:22:44,162 - AnimateDiff - INFO - Loading motion module mm_sd15_v3.safetensors from D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\model\mm_sd15_v3.safetensors
2024-04-10 11:22:44,177 - AnimateDiff - INFO - Guessed mm_sd15_v3.safetensors architecture: MotionModuleType.AnimateDiffV3
2024-04-10 11:22:45,880 - AnimateDiff - INFO - Injecting motion module mm_sd15_v3.safetensors into SD1.5 UNet input blocks.
2024-04-10 11:22:45,880 - AnimateDiff - INFO - Injecting motion module mm_sd15_v3.safetensors into SD1.5 UNet output blocks.
2024-04-10 11:22:45,880 - AnimateDiff - INFO - Setting DDIM alpha.
2024-04-10 11:22:45,880 - AnimateDiff - INFO - Injection finished.
*** Error running before_process: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 776, in before_process
        script.before_process(p, *script_args)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 65, in before_process
        params.set_p(p)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_ui.py", line 172, in set_p
        cn_units = get_controlnet_units(p)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 66, in get_controlnet_units
        cn_unit_dataclass.batch_images = cn_unit_dict.get("batch_images", None)
    AttributeError: 'UiControlNetUnit' object has no attribute 'get'

---
Warning: field infotext in API payload not found in <modules.processing.StableDiffusionProcessingTxt2Img object at 0x0000023970F31000>.
2024-04-10 11:22:45,896 - ControlNet - INFO - unit_separate = False, style_align = False
2024-04-10 11:22:46,209 - ControlNet - INFO - Loading model: control_v11p_sd15_inpaint [ebff9138]
2024-04-10 11:22:46,427 - ControlNet - INFO - Loaded state_dict from [D:\stable-diffusion-webui\stable-diffusion-webui\models\ControlNet\control_v11p_sd15_inpaint.pth]
2024-04-10 11:22:46,427 - ControlNet - INFO - controlnet_default_config
2024-04-10 11:22:47,584 - ControlNet - INFO - ControlNet model control_v11p_sd15_inpaint [ebff9138](ControlModelType.ControlNet) loaded.
*** Error running process: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 784, in process
        script.process(p, *script_args)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1245, in process
        self.controlnet_hack(p)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1230, in controlnet_hack
        self.controlnet_main_entry(p)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 916, in controlnet_main_entry
        input_image, resize_mode = Script.choose_input_image(p, unit, idx)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 700, in choose_input_image
        raise ValueError("controlnet is enabled but no input image is given")
    ValueError: controlnet is enabled but no input image is given

---
  0%|                                                                                            | 0/1 [00:00<?, ?it/s]*** Error executing callback cfg_denoiser_callback for D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\script_callbacks.py", line 230, in cfg_denoiser_callback
        c.callback(params)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_infv2v.py", line 102, in animatediff_on_cfg_denoiser
        ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
    AttributeError: 'NoneType' object has no attribute 'multi_cond'

---
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.97s/it]
*** Error running postprocess_batch_list: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 832, in postprocess_batch_list
        script.postprocess_batch_list(p, pp, *script_args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 84, in postprocess_batch_list
        params.prompt_scheduler.save_infotext_img(p)
    AttributeError: 'NoneType' object has no attribute 'save_infotext_img'

---
*** Error running postprocess: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 816, in postprocess
        script.postprocess(p, processed, *script_args)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 94, in postprocess
        params.prompt_scheduler.save_infotext_txt(res)
    AttributeError: 'NoneType' object has no attribute 'save_infotext_txt'

---
Total progress: 100%|████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.45it/s]
Total progress: 100%|████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.45it/s]

Additional information

No response

Chi-yuki commented 7 months ago

当我使用 stable-diffusion-webui-forge 时,仍然报错:

webui版本

版本: f0.0.17v1.8.0rc-latest-276-g29be1da7  •  python: 3.10.6  •  torch: 2.1.2+cu121  •  xformers: 0.0.23.post1  •  gradio: 3.41.2  •  checkpoint: 86618977ac

animatediff版本

forge/master | b20f7519

调用代码


import requests
import json

url = "http://127.0.0.1:7890/sdapi/v1/txt2img"

payload = {
    "alwayson_scripts": {
    "AnimateDiff": {
      "args": [
        {
          "model": "mm_sd15_v3.safetensors",
          "format": ["PNG", "MP4"],
          "enable": True,
          "video_length": 2,
          "fps": 2,
          "closed_loop": "N",
          "batch_size": 2,
          "video_path": "D:\\frame",
          "mask_path": "D:\\mask"
        }
      ]
    },
    "controlnet": {
      "args": [
        {
            "control_mode": "Balanced", 
            "enabled": True, 
            "generated_image": None, 
            "guidance_end": 1, 
            "guidance_start": 0, 
            "hr_option": "Both", 
            "image": None, 
            "input_mode": "batch", 
            "model": "control_v11p_sd15_inpaint [ebff9138]", 
            "module": "inpaint_only", 
            "pixel_perfect": True, 
            "processor_res": 0.5, 
            "resize_mode": "Crop and Resize", 
            "save_detected_map": True, 
            "threshold_a": 0.5, 
            "threshold_b": 0.5, 
            "use_preview_as_input": False, 
            "weight": 1
        }
      ]
      }
    },
    "batch_size": 8,
    "cfg_scale": 7,
    "denoising_strength": 0.7,
    "height": 1280,
    "prompt": "beautiful_lighting,1girl,",
    "sampler_name" : "Euler a",
    "seed" : -1,
    "steps" : 1,
    "width" : 720
}

response = requests.post(url, json=payload)

Console logs

2024-04-10 21:12:26,805 - AnimateDiff - INFO - AnimateDiff process start.
*** Error running before_process: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 795, in before_process
        script.before_process(p, *script_args)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 63, in before_process
        params.set_p(p)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_ui.py", line 161, in set_p
        cn_units = get_controlnet_units(p)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 62, in get_controlnet_units
        cn_units_dataclass = [ControlNetUnit.from_dict(cn_unit_dict) for cn_unit_dict in cn_units]
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 62, in <listcomp>
        cn_units_dataclass = [ControlNetUnit.from_dict(cn_unit_dict) for cn_unit_dict in cn_units]
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\lib_controlnet\external_code.py", line 256, in from_dict
        **{k: v for k, v in d.items() if k in vars(ControlNetUnit)}
    AttributeError: 'ControlNetUnit' object has no attribute 'items'

---
Warning: field infotext in API payload not found in <modules.processing.StableDiffusionProcessingTxt2Img object at 0x000001FE09ADDDE0>.
2024-04-10 21:12:26,812 - ControlNet - INFO - ControlNet Input Mode: batch
*** Error running process: D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 803, in process
        script.process(p, *script_args)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 548, in process
        self.process_unit_after_click_generate(p, unit, params, *args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 298, in process_unit_after_click_generate
        input_list, resize_mode = self.get_input_data(p, unit, preprocessor, h, w)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 203, in get_input_data
        resize_mode = external_code.resize_mode_from_value(p.resize_mode)
    AttributeError: 'StableDiffusionProcessingTxt2Img' object has no attribute 'resize_mode'

---
2024-04-10 21:12:27,105 - AnimateDiff - INFO - Setting DDIM alpha.
2024-04-10 21:12:27,106 - AnimateDiff - INFO - Injecting motion module mm_sd15_v3.safetensors into SD1.5 UNet.
*** Error running process_before_every_sampling: D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 835, in process_before_every_sampling
        script.process_before_every_sampling(p, *script_args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 555, in process_before_every_sampling
        self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
    KeyError: 0

---
To load target model BaseModel
To load target model MotionWrapper
Begin to load 2 models
Reuse 1 loaded models
Reuse 1 loaded models
[Memory Management] Current Free GPU Memory (MB) =  19894.94873046875
[Memory Management] Model Memory (MB) =  0.0
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  18870.94873046875
[Memory Management] Current Free GPU Memory (MB) =  19894.94873046875
[Memory Management] Model Memory (MB) =  0.0
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  18870.94873046875
Moving model(s) has taken 0.01 seconds
  0%|                                                                                            | 0/1 [00:00<?, ?it/s]*** Error executing callback cfg_denoiser_callback for D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\script_callbacks.py", line 233, in cfg_denoiser_callback
        c.callback(params)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_infv2v.py", line 89, in animatediff_on_cfg_denoiser
        ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
    AttributeError: 'NoneType' object has no attribute 'multi_cond'

---
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.18it/s]
*** Error running postprocess_batch_list: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 859, in postprocess_batch_list
        script.postprocess_batch_list(p, pp, *script_args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 85, in postprocess_batch_list
        params.prompt_scheduler.save_infotext_img(p)
    AttributeError: 'NoneType' object has no attribute 'save_infotext_img'

---
*** Error running postprocess_batch_list: D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 859, in postprocess_batch_list
        script.postprocess_batch_list(p, pp, *script_args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 561, in postprocess_batch_list
        self.process_unit_after_every_sampling(p, unit, self.current_params[i], pp, *args, **kwargs)
    KeyError: 0

---
*** Error running postprocess: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 843, in postprocess
        script.postprocess(p, processed, *script_args)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 90, in postprocess
        params.prompt_scheduler.save_infotext_txt(res)
    AttributeError: 'NoneType' object has no attribute 'save_infotext_txt'

---
Total progress: 100%|████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.58it/s]
Total progress: 100%|████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.59it/s]

forge/master b20f7519

yuchen1984 commented 7 months ago

This PR should fix the problem https://github.com/continue-revolution/sd-webui-animatediff/pull/480