Basic ControlNet support

[monstruosoft]

This PR adds basic ControlNet support to LCM and LCM-LoRA modes. Instead of using a new CLI argument to enable ControlNet, you can enable it with a _--customsettings JSON file or by dynamically creating the corresponding controlnet object in the _lcm_diffusionsetting. Currently, only the initial conditioning scale and control image are set but the GUI versions could edit those values dynamically.

ControlNet is used via the AutoPipelineForText2Image, which means that, if you don't enable ControlNet, a normal StableDiffusion pipeline is created and then FastSD CPU should work just as before.

GUI versions could dynamically create the ControlNet settings. You can currently test ControlNet for the interactive CLI with a JSON file containing something like this:

{
  "controlnet": [{
    "path": "/home/monstruosoft/control_scribble-fp16.safetensors",
    "weight": 0.5,
    "enabled": true,
    "image": "/home/monstruosoft/lena.png"
  }]
}

As an example, the following are images generated using a ControlNet conditioning scale of 1.0:

With a conditioning scale of 0.5:

ControlNet should also work with LoRAs; for example, the following are images generated using the same prompt without LoRA and with the Jane LoRA:

[monstruosoft]

Please note that there's no image preprocessor for ControlNet and you have to provide FastSD CPU a control image with the appropriate format, for example, white traces on a black background for the canny and scribble ControlNet models.

ControNet settings in the JSON file appear as an array because it might be possible to add multi ControlNet support in the future.

Currently, if you don't want to enable ControlNet for the interactive CLI, you can set enabled to false in the JSON file:

{
  "controlnet": [{
    "path": "/home/monstruosoft/control_scribble-fp16.safetensors",
    "weight": 0.5,
    "enabled": false,
    "image": "/home/monstruosoft/lena.png"
  }]
}

The following should also work if you don't want to enable ControlNet:

{
  "controlnet": []
}

As well as:

{
  "controlnet": null
}

Or passing a JSON file with no controlnet key-value pair, or simply not passing a JSON file at all.

Currently, ControlNet should also work in img2img mode.

[monstruosoft]

Thanks for the code review, will work on it.

[rupeshs]

Thanks for the code review, will work on it.

Thanks, please ensure that nothing breaks. We don't want to break anything working in the stable version.

[monstruosoft]

The most recent commit moves the ControlNet code to its own file. Also, you can now dynamically set the ControlNet conditioning scale and control image in the interactive CLI. You can't currently enable/disable ControlNet dynamically; may I suggest having a _rebuildpipeline setting? It might be useful for flagging any action that requires rebuilding the pipeline, so that you don't have to add yet another condition to the pipeline initialization code, for example, it would be easier to have:

if lcm_diffusion_setting.rebuild_pipeline:
  # rebuild pipeline

Than having to add yet another condition like:

if ....
  or ...
  or ...
  or (
    lcm_diffusion_setting.controlnet != None 
    and lcm_diffusion_setting.controlnet.enabled != previous_controlnet_value
  ):
  # rebuild pipeline

Also the most recent commit disables both LoRA and ControlNet at program start for both the GUI and WebUI versions to avoid conflicting settings read from the settings file.

[monstruosoft]

The most recent commit adds ControlNet support to the WebUI. I tried to make the WebUI work the best I could but keep in mind I'm not familiar with Gradio. Still, I think it works fine; I tried a lot of combinations and modes with both LCM and LCM-LoRA models and I think nothing broke, let me know if you find any issues.

Note that placing all ControlNet WebUI controls in a single row is intentional since it might be possible to add multi ControlNet support at a later time (technically it should be already possible at the cost of higher RAM and slower generation time).

Oh, I left the ControlNet models folder currently pointing to the same as the LoRA models folder until it's decided what folder structure should be used.

We don't want to break anything working in the stable version.

Should I make PRs to another branch instead of main for testing?.

[rupeshs]

The most recent commit adds ControlNet support to the WebUI. I tried to make the WebUI work the best I could but keep in mind I'm not familiar with Gradio. Still, I think it works fine; I tried a lot of combinations and modes with both LCM and LCM-LoRA models and I think nothing broke, let me know if you find any issues.

Note that placing all ControlNet WebUI controls in a single row is intentional since it might be possible to add multi ControlNet support at a later time (technically it should be already possible at the cost of higher RAM and slower generation time).

Oh, I left the ControlNet models folder currently pointing to the same as the LoRA models folder until it's decided what folder structure should be used.

We don't want to break anything working in the stable version.

Should I make PRs to another branch instead of main for testing?.

Thanks @monstruosoft , Where to put controlnet models? webui is not mentioning that, it will be helpful for users Regarding multi ControlNet, we don't consider this as of now, we will start with basic implementation.

"Oh, I left the ControlNet models folder currently pointing to the same as the LoRA models folder until it's decided what folder structure should be used." we can use a new folder "controlnet_models" where we will place controlnet models.

[monstruosoft]

OK, will make the suggested change to the ControlNet models folder.

[monstruosoft]

Most recent commit points the default folder for ControlNet models to _controlnetmodels, you may need to delete your settings file for the new defaults to appear. Also, now you can enable/disable ControlNet in the interactive CLI, previously you could only set it once at program start.

[rupeshs]

Most recent commit points the default folder for ControlNet models to _controlnetmodels, you may need to delete your settings file for the new defaults to appear. Also, now you can enable/disable ControlNet in the interactive CLI, previously you could only set it once at program start.

@monstruosoft Thanks , Seems like omegaconf is required for controlnet, we need to add it to requirements.txt

[rupeshs]

@monstruosoft It is raising error when I tried to load https://huggingface.co/lllyasviel/sd-controlnet-canny/blob/main/diffusion_pytorch_model.safetensors

[monstruosoft]

Can you try with the ControlNet models from this page? These are the ones I've been using since they're smaller than the full models. I'm using the medium size models (723 MB) as described here.

Oh, and yes, I do have omegaconf installed, I don't know where it came from.

[rupeshs]

@monstruosoft , Following things noted : 1.Seems like omegaconf is required for controlnet, we need to add it to requirements.txt. omegaconf==2.3.0

2. If ControlNet conditioning scale =1.0 it will raise error

[monstruosoft]

That error seems to be related to the Gradio slider value, will take a look into it.

[monstruosoft]

Should be fixed now.

[rupeshs]

omegaconf Diffusers single file workflow needs to omegaconf

[rupeshs]

@monstruosoft Thanks for the nice work.