InteRACT: Transformer Models for Human Intent Prediction Conditioned on Robot Actions

Kushal Kedia, Atiksh Bhardwaj, Prithwish Dan, Sanjiban Choudhury

Cornell University

ICRA 2024

Installation

Follow these steps to install InteRACT:

Create and activate the conda environment:

cd interact
conda create --name interact python=3.8.16
conda activate interact
pip install -r requirements.txt
pip install -e .

Preliminaries

Set the base_dev_dir to your working directory in all of the config files

Create a new directory for data under interact so that the repo has the following structure:

Repo Structure

├── config
│   ├── *.yaml files 
├── interact
|   ├── checkpoints
|     ├── HH_checkpoints
|     ├── HR_checkpoints
|   ├── data
|     ├── cmu_mocap
|     ├── comad_data
|     ├── comad_hr
|     ├── mapping
|     ├── synthetic_amass
|   ├── model
|     ├── model architecture files...
|   ├── utils
|     ├── utility files...
|   ├── scripts
|     ├── eval_hh.py / eval_hr.py        <- evaluation scripts
|     ├── pretrain_intent_forecaster.py  <- pretraining on H-H 
|     |── finetune_intent_forecaster.py  <- finetuning on H-H
|     |── hr_transfer.py                 <- transferring to H-R
|
├── environment.yml
├── README.md
├── setup.py

Download the data from this link Data

Training

Run the pretraining script on large-scale H-H data:
```
python scripts/pretrain_intent_forecaster.py
```
Run the finetuning script on H-H interaction data.
```
python scripts/finetune_intent_forecaster.py
```
Run the script to transfer the model to the H-R setting.
```
python scripts/hr_transfer.py
```

Evaluation

Run the evaluation script for H-H:
```
python scripts/eval_hh.py
```
Run the evaluation script for H-R:
```
python scripts/eval_hr.py
```

BibTeX

   @article{kedia2023interact,
    title={InteRACT: Transformer Models for Human Intent Prediction Conditioned on Robot Actions},
    author={Kedia, Kushal and Bhardwaj, Atiksh and Dan, Prithwish and Choudhury, Sanjiban},
    journal={arXiv preprint arXiv:2311.12943},
    year={2023}
  }

Acknowledgement

MRT is adapted from MRT

portal-cornell / interact

readme