Project Page • Arxiv Paper • Demo Video • Web Gradio Demo • Citation
Thank Hugging Face🤗 for providing us GPU! Feel free to exprience our online web demo!
conda create -n syntalker python=3.12
conda activate syntalker
pip install -r requirements.txt
bash demo/install_mfa.sh
gdown https://drive.google.com/drive/folders/1tGTB40jF7v0RBXYU-VGRDsDOZp__Gd0_?usp=drive_link -O ./ckpt --folder
gdown https://drive.google.com/drive/folders/1MCks7CMNBtAzU2XihYezNmiGT_6pWex8?usp=drive_link -O ./datasets/hub --folder
For evaluation and training, not necessary for running a web demo or inference.
We provide two ways for getting data, if you want to quickly run this project, you can choose Download the parsed data directly
, or if you want to build datasets from raw data, you can choose Download the original raw data
.
Download the parsed data directly
gdown https://drive.google.com/drive/folders/15gjxrnDQAx2qn7abctYsEx_-WsWH95tz?usp=drive_link -O ./datasets/beat_cache/beat_smplx_en_emage_2_128 --folder
Download the original raw data
bash bash_raw_cospeech_download.sh
python demo.py -c ./configs/diffusion_rvqvae_128_hf.yaml
Notice:
If you use ssh to conect and run code in a headless computer, you may encounter an error pyglet.canvas.xlib.NoSuchDisplayException: Cannot connect to "None"
.
Here, we recommend a method to solve it.
sudo apt-get install libegl1-mesa-dev libgles2-mesa-dev
PYOPENGL_PLATFORM='egl' python demo.py -c ./configs/diffusion_rvqvae_128_hf.yaml
Require download dataset
python test.py -c configs/diffusion_rvqvae_128.yaml
Following EMAGE, you can download SMPLX blender addon, and install it in your blender 3.x or 4.x. Click the button Add Animation
to visualize the generated smplx file (like xxx.npz).
Well, if your multiple gpus, we can parellel run these three commands.
python rvq_beatx_train.py --batch-size 256 --lr 2e-4 --total-iter 300000 --lr-scheduler 200000 --nb-code 512 --code-dim 512 --down-t 2 --depth 3 --dilation-growth-rate 3 --out-dir outputs/rvqvae --vq-act relu --quantizer ema_reset --loss-vel 0.5 --recons-loss l1_smooth --exp-name RVQVAE --body_part upper
python rvq_beatx_train.py --batch-size 256 --lr 2e-4 --total-iter 300000 --lr-scheduler 200000 --nb-code 512 --code-dim 512 --down-t 2 --depth 3 --dilation-growth-rate 3 --out-dir outputs/rvqvae --vq-act relu --quantizer ema_reset --loss-vel 0.5 --recons-loss l1_smooth --exp-name RVQVAE --body_part hands
python rvq_beatx_train.py --batch-size 256 --lr 2e-4 --total-iter 300000 --lr-scheduler 200000 --nb-code 512 --code-dim 512 --down-t 2 --depth 3 --dilation-growth-rate 3 --out-dir outputs/rvqvae --vq-act relu --quantizer ema_reset --loss-vel 0.5 --recons-loss l1_smooth --exp-name RVQVAE --body_part lower_trans
python train.py -c configs/diffusion_rvqvae_128.yaml
Thanks to EMAGE, DiffuseStyleGesture, MDM, T2M-GPT, MoMask, MotionCLIP, TMR, OpenTMA, HumanML3D, human_body_prior, our code is partially borrowing from them. Please check these useful repos.
If you find our code or paper helps, please consider citing:
@inproceedings{chen2024syntalker,
author = {Bohong Chen and Yumeng Li and Yao-Xiang Ding and Tianjia Shao and Kun Zhou},
title = {Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generation},
booktitle = {Proceedings of the 32nd ACM International Conference on Multimedia},
year = {2024},
publisher = {ACM},
address = {New York, NY, USA},
pages = {10},
doi = {10.1145/3664647.3680847}
}