Will controlnet be supported in the future?

[northfoxz]

Hello,

Will controlnet be supported in the future? (for 1.5 and SDXL)

Thank you!

[contentis]

It's definitely on top of my list of features to implement. But currently, I cannot give an ETA.

As ControlNet is not natively supported in Auto1111, what would be the preferred way of integration?

[tydia]

One way I'm thinking of is to first determine whether controlnet extension is installed, if not, abort. If it is present, list all controlnet models and only keep supported models for users to select and convert.

[contentis]

AFAIK there are multiple controlnet extensions out there. Ideally, I don't want to support each one of them, is there a most popular / default one?

[maxious]

https://github.com/Mikubill/sd-webui-controlnet

[tydia]

Yeah I agree with @maxious. Specifically, I believe the following controlnets trained by the author of controlnet are mostly used in sd1.5 and all have same architecture:

control_v11p_sd15_canny control_v11p_sd15_mlsd control_v11f1p_sd15_depth control_v11p_sd15_normalbae control_v11p_sd15_seg control_v11p_sd15_inpaint control_v11p_sd15_lineart control_v11p_sd15s2_lineart_anime control_v11p_sd15_openpose control_v11p_sd15_scribble control_v11p_sd15_softedge control_v11e_sd15_shuffle control_v11e_sd15_ip2p control_v11f1e_sd15_tile

For SDXL, there are just too many variations to consider. Hence I think starting with supporting sd1.5 controlnet models would be a good idea.

[contentis]

When you say variation, do you mean weights?

[tydia]

No I mean all third-party/community control models, which sometimes have different architecture compared to the original controlnets listed above.

FYI, here are some of them: https://huggingface.co/lllyasviel/sd_control_collection

[anwoflow]

Exactly, SDXL CN is a little bit of a mess, unlike with 1.5 you have several models for pose, depth etc. and on top of it, some are ControlNet LoRA models, which is yet again working differently than T2I models that are also available via CN extension ... just like IPAdapter.

Overall it would be fantastic if all of them would be supported, but the number of combinations is huge here. At this point, I'm not sure if working directly of the meta data for each 'job' from Automatic wouldn't be a better source of info, rather than checking which extension is installed, with which model/preprocessor variant. Along the lines "do the same job as THIS image", as it lists all the models that were used at least.

[kanbol]

AFAIK there are multiple controlnet extensions out there. Ideally, I don't want to support each one of them, is there a most popular / default one?

i think openpose is the most popular

[Jonseed]

AFAIK there are multiple controlnet extensions out there. Ideally, I don't want to support each one of them, is there a most popular / default one?

i think openpose is the most popular

And tile (control_v11f1e_sd15_tile) for upscaling!

[mykeehu]

And depth for unknown things

[MattJeanes]

I believe they are referring to the actual SD extension, not the different modes of ControlNet

As mentioned previously the most popular one by far is https://github.com/Mikubill/sd-webui-controlnet

[emperor1412]

I used softedge_hed and IPAdapter in SD1.5 all the time, it would be nice if they can be supported

[psykokwak-com]

Oh yes. It could be great. I use inpaint and openpose.

[lhucklen]

Can this still be a thing please?

[banjaminicc]

There have been various attempts at ControlNet by 3rd parties, I wonder if they'll be of use to everyone here?

https://www.reddit.com/r/StableDiffusion/s/f2NU603gqj

https://github.com/hnsywangxin/controlnet_stable_tensorrt

https://www.cnblogs.com/wildkid1024/p/17668570.html https://www.cnblogs.com/wildkid1024/p/17747027.html

https://tianchi.aliyun.com/forum/post/569424

[wulouzhu]

Looking forward to it

[henbucuoshanghai]

Will controlnet be supported now?

[lix19937]

Ref https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT, then modify.