This is an implementation of Robots learn social skills: End-to-end learning of co-speech gesture generation for humanoid robots (Paper, Project Page)
The original paper used TED dataset, but, in this repository, we modified the code to use Talking With Hands 16.2M for GENEA Challenge 2022. The model is also changed to estimate rotation matrices for upper-body joints instead of estimating Cartesian coordinates.
The code was developed using python 3.8 on Ubuntu 18.04. Pytorch 1.5.0 was used.
Install dependencies
pip install -r requirements.txt
Download the FastText vectors from here and put crawl-300d-2M-subword.bin
to the resource folder (resource/crawl-300d-2M-subword.bin
).
Make LMDB
cd scripts
python twh_dataset_to_lmdb.py [PATH_TO_DATASET]
Update paths and parameters in config/seq2seq.yml
and run train.py
python train.py --config=../config/seq2seq.yml
Do training or use a pretrained model (output/train_seq2seq/baseline_icra19_checkpoint_100.bin
). When you use the pretrained model, please put vocab_cache.pkl file into lmdb train path.
Inference. Output a BVH motion file from speech text (TSV file).
python inference.py [PATH_TO_MODEL_CHECKPOINT] [PATH_TO_TSV_FILE]
Result video for val_2022_v1_006.tsv
by using the challenge visualization server.
https://user-images.githubusercontent.com/9062897/172342581-2dd5a529-5de2-438b-9454-48e7b397ed24.mp4
target_joints
variable in twh_dataset_to_lmdb.py
. Please update data_mean
and data_std
in the config file if you change target_joints
. You can find data mean and std values in the console output of the step 3 (Make LMDB) above.Please see LICENSE.md
@INPROCEEDINGS{
yoonICRA19,
title={Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots},
author={Yoon, Youngwoo and Ko, Woo-Ri and Jang, Minsu and Lee, Jaeyeon and Kim, Jaehong and Lee, Geehyuk},
booktitle={Proc. of The International Conference in Robotics and Automation (ICRA)},
year={2019}
}