[Question] How to actually use ControlNet?

[ke1ne]

Hi, I downloaded a canny model. Could you please give some giude, how to use it in real world? In UI I got this:

Thanks!

[axodox]

Ah, ok there was a mistake in the code, that made UI weird in case of only one controlnet mode was available. I pushed a fix for it.

You otherwise it would be like this:

• You will need a condition image, this could be already a canny edge image, or it can convert it for you. You can either generate such image, or drag and drop it from another app. I think there is an exception if you drop an unsupported file format, I will fix that next weekend.
• Normally the last image is used as input image, but you can also open up the input image panel (with the image button on the left) to use another one.
• Once you have the image enable the control net, select your mode. And if your image is not in the input format enable auto-conversion as well.
• You can adjust the conditioning scale currently it defaults to 0.5, but I will probably change the default to 0.8, as 0.5 does not conform so much to the condition yet.
• You can adjust the denoising strength: a denosing strength < 1 will make this an img2img operation, where the image is used both as one to modify and to extract features from for the controlnet condition. If denoising strength is 1 then you generate image from scratch with using the condition only.
• If you use a mask, then it will become controlnet based inpainting.

[axodox]

The fixed version is released. Currently only canny, depth, hed and openpose have feature extractors, so all others need an input image already in the target format.

[axodox]

I plan to do more bugfixing and documentation on the weekend. Also writing guides for controlnet ready model conversion.

[ke1ne]

Thank you for the update. Also, that guide could be useful in the Wiki. Cheers!