search

Mikubill / sd-webui-controlnet

WebUI extension for ControlNet
GNU General Public License v3.0
17.1k stars 1.97k forks source link

Support union ControlNet #2988

Closed huchenlei closed 4 months ago

huchenlei commented 4 months ago

Closes https://github.com/Mikubill/sd-webui-controlnet/issues/2986

https://huggingface.co/xinsir/controlnet-union-sdxl-1.0

Openpose

image 00021-1413879340

Do not use key map with face and hand as the union controlnet does not seem to be trained with hand/face annotation image 00020-1413879340

Canny

image 00017-1413879340

Depth

image 00018-1413879340

SoftEdge

image 00019-1413879340

Normal

image 00022-1413879340

Segmentation

Note: ofade20k preprocessor does not seem to work image image

Multi-ControlNet

Currently the multi-controlnet is not working in the optimal way described in the original paper, but you can still try use it, as it can help you save VRAM by avoid loading another controlnet for different type of control.

Unit1 setup

image

Unit 2 setup

image

Generation result

00026-1413879340

Appendix

Input images image (1) input

Mask image

Generation parameters (Parameters including seed are fixed for all examples)

1girl, sitting on throne,, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 30, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 1413879340, Size: 1024x1024, Model hash: 5d973c1f8a, Model: animagine-xl-2.0, ControlNet 0: "Module: none, Model: controlnet++_union_sdxl [15e6ad5d], Weight: 1.0, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.0, Guidance End: 1.0, Pixel Perfect: False, Control Mode: Balanced", ControlNet 1: "Module: canny, Model: controlnet++_union_sdxl [15e6ad5d], Weight: 1.0, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 100.0, Threshold B: 200.0, Guidance Start: 0.0, Guidance End: 1.0, Pixel Perfect: False, Control Mode: Balanced", Version: v1.9.4-249-g93c00b2a
FurkanGozukara commented 3 months ago

what should be the file name to use this? when downloading from hugging face

he also has canny model too what file name i should do?