mit-biomimetics / fld

Repository for our paper: FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning, Proceedings of the 12th International Conference on Learning Representations (ICLR)
https://sites.google.com/view/iclr2024-fld/home
Other
245 stars 27 forks source link

FLD with MIT Humanoid

This repository provides the Fourier Latent Dynamics (FLD) algorithm that represents high-dimension, long-horizon, highly nonlinear, period or quasi-period data in a continuously parameterized space. This work demonstrates its representation and generation capability with a robotic motion tracking task on MIT Humanoid using NVIDIA Isaac Gym.

fld

Paper: FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning
Project website: https://sites.google.com/view/iclr2024-fld/home

Maintainer: Chenhao Li
Affiliation: Biomimetic Robotics Lab, Massachusetts Institute of Technology
Contact: chenhli@mit.edu

Installation

  1. Create a new python virtual environment with python 3.8

  2. Install pytorch 1.10 with cuda-11.3

    pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
  3. Install Isaac Gym

    • Download and install Isaac Gym Preview 4

      cd isaacgym/python
      pip install -e .
    • Try running an example

      cd examples
      python 1080_balls_of_solitude.py
    • For troubleshooting, check docs in isaacgym/docs/index.html

  4. Install humanoid_gym

    git clone https://github.com/mit-biomimetics/fld.git
    cd fld
    pip install -e .

Configuration

Usage

FLD Training

python scripts/fld/experiment.py

FLD Evaluation

python scripts/fld/evaluate.py

Motion Visualization

python scripts/fld/preview.py

Policy Training

python scripts/train.py --task mit_humanoid

Policy Playing

python scripts/play.py --load_run "<date_time>_<run_name>"

Troubleshooting

RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)

Known Issues

The ALPGMMSampler utilizes faiss for efficient similarity search and clustering of dense vectors in the latent parameterization space. The installation of faiss requires a compatible CUDA version. The current implementation is tested with faiss-cpu and faiss-gpu with cuda-10.2.

Citation

@article{li2024fld,
  title={FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning},
  author={Li, Chenhao and Stanger-Jones, Elijah and Heim, Steve and Kim, Sangbae},
  journal={arXiv preprint arXiv:2402.13820},
  year={2024}
}

References

The code is built upon the open-sourced Periodic Autoencoder (PAE) Implementation, Isaac Gym Environments for Legged Robots and the PPO implementation. We refer to the original repositories for more details.