Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots.

A PyTorch implementation of "Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots".

@article{yoon2018robots,
  title={Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots},
  author={Yoon, Youngwoo and Ko, Woo-Ri and Jang, Minsu and Lee, Jaeyeon and Kim, Jaehong and Lee, Geehyuk},
  journal={arXiv preprint arXiv:1810.12541},
  year={2018}
}

[Paper]

You can find the dataset and more information on the paper here.

Update 28/10

Thank you Eddie Wang for updating the implementation!

Update 03/02/2020

Have a look at this repository for an up to date implementation.

Installation

The code is implemented in Python 3.7 and PyTorch 1.0

The implementation in this repository is based on the pre-print as it appeared on arXiv. I will update the implementation when more information becomes available. Please create an issue when you spot bugs.

Check out this repository for translating 2D frontal pose to 3D frontal pose.

Running the code

Please follow the following steps to run the code:

1- Create the following directories:

./data/shots/JSON/
./data/videos_tedx_subtitles/
./data/shots/CSV/
./models/

2- Place the youtube-gestures-dataset generated .csv, .json, and .vtt in their respective foldrs. Please take care to make sure that file names are formated exactly as .ext to avoid file errors.

3- Run preprocessing.py to generate the pickle file "preprocessed_1295videos.pickle".

4- Move the file generated in step 3 to the ./data directory

5- If running for the first time, modify main.py so that the parameter pre_trained_file=None

6- Run main.py