MusePose

MusePose is an image-to-video generation framework for virtual human under control signal such as pose.

MusePose is the last building block of the Muse opensource serie. Together with MuseV and MuseTalk, we hope the community can join us and march towards the vision where a virtual human can be generated end2end with native ability of full body movement and interaction. Please stay tuned for our next milestone!

Comfyui-MusePose

If you're running on Linux, or non-admin account on windows you'll want to ensure /ComfyUI/custom_nodes and Comfyui-MusePose has write permissions.

Followed ComfyUI's manual installation steps and do the following:

• Navigate to your /ComfyUI/custom_nodes/ folder
• Run git clone https://github.com/TMElyralab/Comfyui-MusePose.git

• Navigate to your /ComfyUI/custom_nodes/Comfyui-MusePose folder and run

  pip install -r requirements.txt
  
  pip install --no-cache-dir -U openmim 
  mim install mmengine 
  mim install "mmcv>=2.0.1" 
  mim install "mmdet>=3.1.0" 
  mim install "mmpose>=1.1.0" 

• Start ComfyUI

Updates

• requirements.txt: diffusers 0.27.2 is now supported

Download weights

You can download weights manually as follows:

1. Download our trained weights.

2. Download the weights of other components:

   • sd-image-variations-diffusers
   • sd-vae-ft-mse
   • dwpose
   • yolox - Make sure to rename to yolox_l_8x8_300e_coco.pth
   • image_encoder

Finally, these weights should be organized in pretrained_weights as follows:

./pretrained_weights/
|-- MusePose
|   |-- denoising_unet.pth
|   |-- motion_module.pth
|   |-- pose_guider.pth
|   └── reference_unet.pth
|-- dwpose
|   |-- dw-ll_ucoco_384.pth
|   └── yolox_l_8x8_300e_coco.pth
|-- sd-image-variations-diffusers
|   └── unet
|       |-- config.json
|       └── diffusion_pytorch_model.bin
|-- image_encoder
|   |-- config.json
|   └── pytorch_model.bin
└── sd-vae-ft-mse
    |-- config.json
    └── diffusion_pytorch_model.bin

workflow demo

https://github.com/TMElyralab/Comfyui-MusePose/blob/main/musepose-workflow-demo.json

https://github.com/TMElyralab/Comfyui-MusePose/assets/114042542/9cd8b9b8-6876-4281-b7a0-a7fbcb2de7e1