search

deforum-art / sd-webui-deforum

Deforum extension for AUTOMATIC1111's Stable Diffusion webui
https://deforum.github.io
Other
2.69k stars 389 forks source link

ControlNet support #359

Closed kabachuha closed 1 year ago

kabachuha commented 1 year ago

ControlNet is an especially powerful new feature which can be used to guide image generation.

It is going to be extremely useful for Deforum animation creation, so it's top priority to integrate it into Deforum.

Since it's applicable both to txt2img and img2img, it can be feed similarly to video masking. Fortunately, we already have the composable mask mechanism.

Meanwhile, another lucky thing is that someone's doing an API layer on top of the CN extension. If we base Deforum support on top of the functions there, it will greatly speedup the transition process.

https://github.com/Mikubill/sd-webui-controlnet/pull/194/

kabachuha commented 1 year ago

I added all you because you have expertise in video parts insertion. I'm going to push a draft version soon and then we need to make it like the second 'hybrid mode' or something like that

kabachuha commented 1 year ago

The PR is live! (although still a bit raw)

screan commented 1 year ago

after updating, deforum tab is missing and console throwing this error:

Error executing callback ui_tabs_callback for E:\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum.py
Traceback (most recent call last):
  File "E:\stable-diffusion-webui\modules\script_callbacks.py", line 119, in ui_tabs_callback
    res += c.callback() or []
  File "E:\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum.py", line 198, in on_ui_tabs
    components = deforum_args.setup_deforum_setting_dictionary(None, True, True)
  File "E:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\args.py", line 683, in setup_deforum_setting_dictionary
    controlnet_dict = setup_controlnet_ui()
  File "E:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\deforum_controlnet.py", line 171, in setup_controlnet_ui
    return setup_controlnet_ui_raw()
  File "E:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\deforum_controlnet.py", line 89, in setup_controlnet_ui_raw
    from scripts.processor import canny, midas, midas_normal, leres, hed, mlsd, openpose, pidinet, simple_scribble, fake_scribble, uniformer
ImportError: cannot import name 'leres' from 'scripts.processor' (E:\stable-diffusion-webui/extensions/sd-webui-controlnet\scripts\processor.py)
kabachuha commented 1 year ago

@screan disabled leres in https://github.com/deforum-art/deforum-for-automatic1111-webui/commit/147b2cbbefb8fdcfa0b3494e7bc489a9febf70f7. Try launching now

screan commented 1 year ago 
Error executing callback ui_tabs_callback for E:\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum.py
Traceback (most recent call last):
  File "E:\stable-diffusion-webui\modules\script_callbacks.py", line 119, in ui_tabs_callback
    res += c.callback() or []
  File "E:\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum.py", line 198, in on_ui_tabs
    components = deforum_args.setup_deforum_setting_dictionary(None, True, True)
  File "E:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\args.py", line 686, in setup_deforum_setting_dictionary
    controlnet_dict = setup_controlnet_ui()
  File "E:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\deforum_controlnet.py", line 171, in setup_controlnet_ui
    return setup_controlnet_ui_raw()
  File "E:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\deforum_controlnet.py", line 89, in setup_controlnet_ui_raw
    from scripts.processor import canny, midas, midas_normal, hed, mlsd, openpose, pidinet, simple_scribble, fake_scribble, uniformer
ImportError: cannot import name 'pidinet' from 'scripts.processor' (E:\stable-diffusion-webui/extensions/sd-webui-controlnet\scripts\processor.py)
kabachuha commented 1 year ago

Upgrade or Downgrade your ControlNet version to a24089a62e70a7fae44b7bf35b51fd584dd55e25 — it's the one used in the integration development

screan commented 1 year ago

thanks @kabachuha !

kabachuha commented 1 year ago

Glad it helped!

I'll look into other versions support too, but now I need to get some sleep.

@screan, send your results when you get them. I'm eager to see how it works for you 🙂

kabachuha commented 1 year ago

The recommended version is https://github.com/Mikubill/sd-webui-controlnet/archive/c9340671d6d59e5a79fc404f78f747f969f87374.zip now