search

Haidra-Org / hordelib

A wrapper around ComfyUI to allow use by the AI Horde.
https://hordelib.org
GNU Affero General Public License v3.0
56 stars 17 forks source link

ControlNet 1.1+ Compatibility #317

Open dunkeroni opened 1 year ago

dunkeroni commented 1 year ago

The newer ControlNet options available in 1.1 and onwards offer a level of complexity that the Horde is not currently able to utilize. In addition to having more model types in general, there are also preprocessors that do not use a model (e.g. reference_only mode that allows recreating subjects based on an input image) as well as models that do not rely on preprocessors (e.g. experimental Tile model that allows tile-based upscaling without introducing repetition artifacts). New models are architecturally the same as before, so ComfyUI already supports them. Just the Horde is not conducive to adding new CN features in its current state.

At minimum, the inputs for ControlNet operations should be separated configurable parts of the request, and recommended setups can be handled on the frontends. Ideally it should align with whatever plans are in place for the Horde server side to handle future Comfy pipelines.

GenerationInputStable {
  ...,
  params {
    ...,
    ComfyProcess {
      process_type: "ControlNet",
      controlnet_preprocessor: <string>,
      controlnet_model: <string>,
      controlnet_strength: <number>,
      controlnet_img: <encoded webp>
      controlnet_return_control_map: <boolean>
tazlin commented 3 months ago