dvlab-research / ControlNeXt

Controllable video and image Generation, SVD, Animate Anyone, ControlNet, ControlNeXt, LoRA
Apache License 2.0
1.39k stars 66 forks source link

How to reuse DWPose video? #17

Closed SoaringTiger closed 2 months ago

SoaringTiger commented 2 months ago

Can I directly use the DWPose action MP4 as an input parameter, so I can reuse DWPose video without having to regenerate it every time ?

cly2625 commented 2 months ago

Try MimicMotion ?

Pbihao commented 2 months ago

[There is a pose alignment process that first extracts pose information and then maps the source pose video coordinates to the reference image's human pose. For more details, please refer to this https://github.com/dvlab-research/ControlNeXt/blob/main/ControlNeXt-SVD-v2/dwpose/preprocess.py

Certainly, you can directly store the pose coordinates and apply the mapping during inference. This mapping is essential because SVD tends to perform poorly when generating large motion sequences.