New SDXL Controlnet: How to use it?

[Echolink50]

How do we use these on Windows? Anybody got the example images and/or instructions? Thanks

https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0

https://huggingface.co/SargeZT/controlnet-v1e-sdxl-depth

[comfyanonymous]

https://comfyanonymous.github.io/ComfyUI_examples/controlnet/

Use them like regular controlnet models. You can use the image->preprocessor->canny in the base install to process images for the canny controlnet.

[Richaskin]

As a bit of a beginner to this, can anyone help explain step by step how to install ControlNet for SDXL using ComfyUI.

[jteijema]

https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0/discussions/1#64d48d8fe143badbbafb5b43 here's a good example

[Enzzer]

I'm getting this error when trying to load the new diffuser sdxl canny model:

Error occurred when executing ControlNetLoader:

'body.0.block2.weight'

File "/home/admin/projects/ComfyUI/execution.py", line 145, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/home/admin/projects/ComfyUI/execution.py", line 75, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/home/admin/projects/ComfyUI/execution.py", line 68, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/home/admin/projects/ComfyUI/nodes.py", line 536, in load_controlnet
controlnet = comfy.sd.load_controlnet(controlnet_path)
File "/home/admin/projects/ComfyUI/comfy/sd.py", line 751, in load_controlnet
net = load_t2i_adapter(controlnet_data)
File "/home/admin/projects/ComfyUI/comfy/sd.py", line 899, in load_t2i_adapter
ksize = t2i_data['body.0.block2.weight'].shape[2]

This happens for both of the controlnet model loaders (regular and diff).

[Kelier]

I can handle it. Is there anyone who could help me fix this?

[comfyanonymous]

Make sure you update your comfyui to the latest and use make sure the controlnet is compatible with the model being used.

[cardamon]

FYI you can find SDXL compatible ControlNet checkpoints here: https://huggingface.co/models?other=stable-diffusion-xl&other=controlnet

E.g. https://huggingface.co/thibaud/controlnet-openpose-sdxl-1.0/tree/main is a working OpenPose ControlNet.

[AtomicPerception]

Depth and ZOE depth are named the same. "diffusion_pytorch_model.safetensors" Where do I place these files? I can't just copy them into the ComfyUI\models\controlnet folder. They'll overwrite one another.

Shouldn't they have unique names?

[ltdrdata]

Depth and ZOE depth are named the same. "diffusion_pytorch_model.safetensors" Where do I place these files? I can't just copy them into the ComfyUI\models\controlnet folder. They'll overwrite one another.

Shouldn't they have unique names?

Make subfolder and save it to there.

[ghostsquad]

FYI you can find SDXL compatible ControlNet checkpoints here: https://huggingface.co/models?other=stable-diffusion-xl&other=controlnet

E.g. https://huggingface.co/thibaud/controlnet-openpose-sdxl-1.0/tree/main is a working OpenPose ControlNet.

What's the differences between the small & mid models?

[cardamon]

Some more recent links which may be useful:

• https://huggingface.co/lllyasviel/sd_control_collection/tree/main
• https://stable-diffusion-art.com/controlnet-sdxl/ (guide is for AUTOMATIC1111 but still very useful for demonstration purposes)

[ASmallPotato]

I think I got the same error as Kelier's. (can't see the last part of the stack trace, so can't really compare is it throwing from the same place, but the error message is the same)

I was using the code few hours before this commit. and said commit fixed the error.