1. VisualTactileGym

Tactile simulation

This project provides tactile simulation on the isaacgym platform and is managed uniformly through point clouds and six dimensional forces, which can be used as input together with camera point clouds.

Custom ( Robot Arm / Gripper / Tactile Sensor )

At the same time, this simulation supports different robot systems

2. Dependencies

Some dependencies can be installed by

pip install -r ./requirements.txt

Isaac Gym

Our framework is implemented on Isaac Gym simulator, the version we used is Preview Release 4. You may encounter errors in installing packages, most solutions can be found in the official docs.

Pointnet2

Install pointnet++ manually.

cd {the dir for packages}
git clone --recursive https://github.com/erikwijmans/Pointnet2_PyTorch
cd Pointnet2_PyTorch

pip install -r requirements.txt
pip install -e .

Finally, run the following to install other packages.

# make sure you are at the repository root directory
pip install -r requirements.txt

Assets（Optional）

Google Drive Link: https://drive.google.com/file/d/16qATmRousv1vgVg3yQnahmRkPLoQ8lzG/view?usp=drive_link

3. Training

Tensorboard logdir :/run

1. RL (without tactile and visual pointcloud)

For env modify this file : cfg/task/xxxx.yaml
numEnvs : 512+

obs_type: ["oracle"]

For algo modify this file: cfg/train/sac/sac_xxxx.yaml
load_iter: when rl model saved

python3 train.py --task xxxx --algo sac --headless

For testing:

python3 train.py --task xxxx --algo sac --headless --test

2. RL or Custom Algo (With tactile or visual pointcloud)

For env modify this file : cfg/task/xxxx.yaml
numEnvs : 16+
obs_type: ["oracle","tactile","pointcloud"]

Other config files are the same as RL

python3 train.py --task xxxx --algo custom --headless

For testing:

python3 train.py --task xxxx --algo custom --test --headless

Reference

https://github.com/PKU-MARL/DexterousHands
https://github.com/hyperplane-lab/RLAfford