Add DWposeDetector

[haofanwang]

DWPose is a newly proposed pose detector that performs much better than OpenPose especially on hand. It can be used with ControlNet-OpenPose model directly.

The only thing is to install some new requirements and download configs&checkpoints.

import torch
from controlnet_aux import DWposeDetector

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
img = Image.open(BytesIO(response.content)).convert("RGB").resize((512, 512))

dwpose = DWposeDetector(det_config, det_ckpt, pose_config, pose_ckpt, device)
processed_image_dwpose = dwpose(img)

[pdoane]

Mediapipe is also handled as an optional processor - you can see how we handled that there.

Maybe should also change the device parameter to the constructor? The other processors have a to method similar to diffusers.

[haofanwang]

Sure. Will revise soon.

[haofanwang]

@pdoane Done. Should be ready to merge now.

[patrickvonplaten]

Thanks a bunch for the PR and review here @haofanwang and @pdoane

[patrickvonplaten]

BTW @pdoane if you want I can make you an admin of this package so that you are not stuck when I'm on holiday (sorry needed a break for like two weeks). Feel free to send me an email to patrick.v.platen[at]gmail.com if you'd like to be added as an admin

[moono]

Hello, I just bumped into this,

I saw the dwpose announced onnx version of dwpose. On their README,

2023/08/07: We release a new DWPose with onnx. You can avoid installing mmcv through this. See branch onnx.

and it seems onnx version is now default branch which can avoid installing mmcv.

So @haofanwang, could you update as onnx version? Which seem to be more popular(?) and easy to install, instead of mmcv.

And @patrickvonplaten, current version of controlnet_aux is 0.0.6 which does not include dwpose. Will you update pypi package so that others can try out?

[AparnaAgrawal02]

hi, is it still in pipeline or can we try this? i tried: pose_config = "dwpose/dwpose-l_384x288.py" det_config = "yolox_l_8xb8-300e_coco.py" det_ckpt = "/scratch/aparna/yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth" pose_ckpt= "/scratch/aparna/dw-ll_ucoco_384.pth" import torch device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') dwpose = DWposeDetector(det_config=det_config, det_ckpt=det_ckpt, pose_config=pose_config, pose_ckpt=pose_ckpt, device=device)

but its currently giving NameError: name 'init_detector' is not defined error

[bb67ao]

hi, is it still in pipeline or can we try this? i tried: pose_config = "dwpose/dwpose-l_384x288.py" det_config = "yolox_l_8xb8-300e_coco.py" det_ckpt = "/scratch/aparna/yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth" pose_ckpt= "/scratch/aparna/dw-ll_ucoco_384.pth" import torch device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') dwpose = DWposeDetector(det_config=det_config, det_ckpt=det_ckpt, pose_config=pose_config, pose_ckpt=pose_ckpt, device=device)

but its currently giving NameError: name 'init_detector' is not defined error

Similar to you，NameError: name 'adapt_mmdet_pipeline' is not defined