search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.34k stars 381 forks source link

[Feature]: ControlNet Omni pipeline #3322

Open lbeltrame opened 2 weeks ago

lbeltrame commented 2 weeks ago

Feature description

This is opened as a feature just to keep track of things.

The recently-released ControlNet Omni model may require a specific Diffusers pipeline (or similar approaches) for inference (https://github.com/xinsir6/ControlNetPlus/tree/main?tab=readme-ov-file#controlmode), but it's not clear (clarification asked: https://huggingface.co/xinsir/controlnet-union-sdxl-1.0/discussions/8).

If it is required, it would be nice to have it in SD.Next as one single ControlNet model would make its use considerably easier. If it is not required, this issue can be simply closed.

Version Platform Description

Latest dev as of 2024-07-09.

lbeltrame commented 2 weeks ago

There's a bunch of code at https://github.com/xinsir6/ControlNetPlus/blob/main/pipeline/pipeline_controlnet_union_sd_xl.py but I have no idea how much this is copied from diffusers and where the "secret sauce" lies.

vladmandic commented 2 weeks ago

i've already added support for new xinsir controlnet union model in dev branch and it works fine with standard controlnet pipeline. the new "enhanced" pipeline is not required, but it is recommended to allow for more granular control - i do plan to add it as well, but its not a blocker for adoption of the new model.