[Feature] Support for SD3 ControlNet

[Fannovel16]

Chads from InstantX (who created InstantID) has made several ControlNet for SD3-Medium, including:

• InstantX/SD3-Controlnet-Canny
• InstantX/SD3-Controlnet-Pose
• InstantX/SD3-Controlnet-Tile
• InstantX/SD3-Controlnet-Inpainting

Their implementation can be found at https://github.com/huggingface/diffusers/pull/8566

[kadirnar]

+1

[sanjay-uzu]

But these models are not working in comfyui

[Fannovel16]

@sanjay-uzu That is why we need an issue

[zsxkib]

"Chads from InstantX" lol @Fannovel16

[geroldmeisinger]

controlnet support merged into diffusers https://github.com/huggingface/diffusers/pull/8566#event-13208383785

[geroldmeisinger]

state of the model seems subpar atm https://huggingface.co/InstantX/SD3-Controlnet-Canny/discussions/4 but I guess a 1.0 version will follow soon anyone had better results yet?

[kijai]

state of the model seems subpar atm https://huggingface.co/InstantX/SD3-Controlnet-Canny/discussions/4 but I guess a 1.0 version will follow soon anyone had better results yet?

Here's some of my tests, post and tile work pretty good. This is using diffusers with a wrapper node I made to test this, I added some of the standard CN features that was missing (start/end, guess_mode). Canny seems to work better with more defined input, I found using lineart preproc better. I do definitely think it's worth implementing!

[al-swaiti]

its here but in diffusers mode https://github.com/ZHO-ZHO-ZHO/ComfyUI-SD3-Medium-CN-Diffusers waiting for normal one

[mcmonkey4eva]

Added to Comfy yesterday https://github.com/comfyanonymous/ComfyUI/commit/264caca20e1e051f3bcbec59d7209513ea99d4ed use ControlNetApplySD3 node