OpenPose only works about 50% of the time

[dayunbao]

Ever since upgrading to Stable Diffusion XL, it's very difficult to get the OpenPose functionality to work. I'd say it only works for me about 50% of the time. There are times when it seems to fail completely - the OpenPose input image that's returned with the generated images is totally black. Other times, I do see the keypoints in the OpenPose input image that's returned with the generated images, but the OpenPose data was totally ignored.

I don't know if this is a problem with the XL OpenPose models people have trained for ControlNet, or a problem with ControlNet itself.

Here is an example of the 2nd case:

I'm using two LoRA models (a character and a style), but OpenPose will sometimes refuse to work even if I'm not using LoRA models.

API request body:

{
    "alwayson_scripts": {
        "API payload": {
            "args": []
        },
        "ControlNet": {
            "args": [
                {
                    "batch_images": "",
                    "control_mode": "Balanced",
                    "enabled": true,
                    "guidance_end": 1,
                    "guidance_start": 0,
                    "image": {
                        "image": "long string"
                    },
                    "input_mode": "simple",
                    "is_ui": true,
                    "loopback": false,
                    "low_vram": true,
                    "model": "kohya_controllllite_xl_openpose_anime_v2 [b0fa10bb]",
                    "module": "openpose",
                    "output_dir": "",
                    "pixel_perfect": true,
                    "processor_res": 512,
                    "resize_mode": "Crop and Resize",
                    "save_detected_map": true,
                    "threshold_a": -1,
                    "threshold_b": -1,
                    "weight": 1
                },
            ]
        },
    },
    "batch_size": 4,
    "cfg_scale": 7,
    "comments": {},
    "disable_extra_networks": false,
    "do_not_save_grid": false,
    "do_not_save_samples": false,
    "enable_hr": false,
    "height": 1024,
    "hr_negative_prompt": "",
    "hr_prompt": "",
    "hr_resize_x": 0,
    "hr_resize_y": 0,
    "hr_scale": 2,
    "hr_second_pass_steps": 0,
    "hr_upscaler": "Latent",
    "n_iter": 1,
    "negative_prompt": "",
    "override_settings": {},
    "override_settings_restore_afterwards": true,
    "prompt": " <lora:keqing3:1> keqing (genshin impact),purple eyes,purple hair,twintails, hair ornament,bare shoulders,dress,hair bun,detached sleeves,  <lora:sdxl:1>",
    "restore_faces": false,
    "s_churn": 0.0,
    "s_min_uncond": 0,
    "s_noise": 1.0,
    "s_tmax": null,
    "s_tmin": 0.0,
    "sampler_name": "DPM++ 3M SDE Exponential",
    "script_args": [],
    "script_name": null,
    "seed": -1,
    "seed_enable_extras": true,
    "seed_resize_from_h": -1,
    "seed_resize_from_w": -1,
    "steps": 50,
    "styles": [],
    "subseed": -1,
    "subseed_strength": 0,
    "tiling": false,
    "width": 1024
}

The generation settings:

<lora:keqing3:1> keqing (genshin impact),purple eyes,purple hair,twintails, hair ornament,bare shoulders,dress,hair bun,detached sleeves, <lora:sdxl:1>
Steps: 50, Sampler: DPM++ 3M SDE Exponential, CFG scale: 7, Seed: 610591763, Size: 1024x1024, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, VAE hash: 235745af8d, VAE: sdxl_vae.safetensors, ControlNet 0: "Module: openpose, Model: kohya_controllllite_xl_openpose_anime_v2 [b0fa10bb], Weight: 1, Resize Mode: Crop and Resize, Low Vram: True, Processor Res: 512, Guidance Start: 0, Guidance End: 1, Pixel Perfect: True, Control Mode: Balanced, Save Detected Map: True", Lora hashes: "keqing3: cba586b84494, sdxl: 2fc5967b49c2", Version: v1.6.0-1-g0562456f

The relevant API logs, which never show any errors:

Nov 25 06:14:53 ip-172-31-42-185.eu-north-1.compute.internal bash[5578]: 2023-11-25 06:14:53,503 - ControlNet - INFO - Loading model from cache: kohya_controllllite_xl_openpose_anime_v2 [b0fa10bb]
Nov 25 06:14:53 ip-172-31-42-185.eu-north-1.compute.internal bash[5578]: 2023-11-25 06:14:53,504 - ControlNet - INFO - Loading preprocessor: openpose
Nov 25 06:14:53 ip-172-31-42-185.eu-north-1.compute.internal bash[5578]: 2023-11-25 06:14:53,504 - ControlNet - INFO - preprocessor resolution = 1024
Nov 25 06:14:53 ip-172-31-42-185.eu-north-1.compute.internal bash[5578]: 2023-11-25 06:14:53,553 - ControlNet - INFO - ControlNet Hooked - Time = 0.21093463897705078

The response:

[dayunbao]

OpenPose is one of the most important parts of ControlNet for me. I use it all the time. If you could find the time to investigate this, I would really appreciate it @lllyasviel!

[dayunbao]

I've been testing with SD 1.5, and OpenPose works every time. So this is definitely related to SD XL, or the OpenPose XL models.