XSkill: Cross Embodiment Skill Discovery

^1,²Mengda Xu, ^1,Zhenjia Xu, ¹Cheng Chi, ^2,³Manuela Veloso, ¹Shuran Song

¹Columbia University, ²JP Morgan AI Research,³CMU

CoRL 2023

This repository contains code for training and evaluating XSkill in both simulation and real-world settings. Teaser Image

🚀 Installation

Follow these steps to install XSkill:

Create and activate the conda environment:

cd xskill
conda env create -f environment.yml
conda activate xskill
pip install -e .

📦 Simulation Dataset

To set up the simulation dataset:

Create a new directory for datasets under XSkill:

mkdir datasets
cd datasets
wget https://xskill.cs.columbia.edu/data/kitchen.zip
unzip kitchen.zip

Set the base_dev_dir config/simulation/create_kitchen_datase.yaml to your working directory. Run the following command to generate the cross-embodiment kitchen data and the training mask:
```
cd scripts
python create_all_kitchen_dataset.py
python extract_kitchen_info.py
```

🌐 Real World Dataset

To Download the real world kitchen dataset:

   mkdir datasets
   cd datasets
   wget https://xskill.cs.columbia.edu/data/real_kitchen_data.zip

🚴‍♂️ Training

Simulation

Run the skill discovery script:
```
python scripts/skill_discovery.py
```
Label the dataset using the learned prototype by the trained model.
```
python scripts/label_sim_kitchen_dataset.py
```
Execute the skill transfer and composing script. Replace the pretrain_path and pretrain_ckpt in cfg/simulation/skill_transfer_composing.yaml
```
python scripts/skill_transfer_composing.py
```
Real World
Execute the real-world skill discovery script:
```
python scripts/realworld/skill_discovery.py
```

Label the real-world dataset:

python scripts/realworld/label_real_kitchen_dataset.py

📊 Visualization

Open the provided Jupyter notebook viz_real.ipynb to visualize the learned prototypes:

BibTeX

   @inproceedings{
        xu2023xskill,
        title={{XS}kill: Cross Embodiment Skill Discovery},
        author={Mengda Xu and Zhenjia Xu and Cheng Chi and  Manuela Veloso and Shuran Song},
        booktitle={7th Annual Conference on Robot Learning},
        year={2023},
        url={https://openreview.net/forum?id=8L6pHd9aS6w}
        }

License

This repository is released under the MIT license.

Acknowledgement

Diffusion Policy is adapted from Diffusion Policy
Many useful utilies are adapted from XIRL.

real-stanford / xskill

readme