search

jr-robotics / robo-gym

An open source toolkit for Distributed Deep Reinforcement Learning on real and simulated robots.
https://sites.google.com/view/robo-gym
MIT License
426 stars 75 forks source link
openai-gym reinforcement-learning-environments robo-gym robotics

readme

robo-gym logo

robo-gym


PyPI - Python Version PyPI

robo-gym is an open source toolkit for distributed reinforcement learning on real and simulated robots.

robo-gym provides a collection of reinforcement learning environments involving robotic tasks applicable in both simulation and real world robotics. Additionally, we provide the tools to facilitate the creation of new environments featuring different robots and sensors.

Main features :

A paper describing robo-gym has been accepted for IROS 2020. A video showcasing the toolkit's capabilities and additional info can be found on our website

NOTE: We are continuously working to improve and expand robo-gym. If you are interested in reproducing the results obtained in the IROS 2020 paper please refer to v.0.1.0 for all the 3 repositories involved in the framework: robo-gym, robo-gym-robot-servers, robo-gym-server-modules.

NOTE: robo-gym is undergoing a necessary overhaul process. Things may break temporarily, and some old setups may not be supported anymore. In particular:

See the News section

Table of Contents

Basics

back to top

The robo-gym framework is composed of several building blocks. Detailed information on them is given here and in the paper.

robo-gym framework

The framework can be subdivided in two main parts:

The Robot Server Side and the Environment Side can run on the same PC or on different PCs connected via network.

Installation

back to top

Environment Side

Requirements: Python >= 3.8

You can perform a minimal install of robo-gym with:

git clone https://github.com/jr-robotics/robo-gym.git
cd robo-gym
pip install -e .

If you prefer, you can do a minimal install of the packaged version directly from PyPI:

pip install robo-gym

Robot Server Side

Requirements: Ubuntu 20.04 (recommended) or 18.04.

The Robot Server Side can be installed on the same machine running the Environment Side and/or on other multiple machines.

Install robo-gym-robot-servers following the instructions in the repository's README.

How to use

back to top

Each environment comes with a version to be run with a simulated version of the robot and the scenario and version to be run with the real robot. Simulated environments have a name ending with Sim whereas real robot environments have a name ending with Rob.

Simulated Environments

Before making a simulated environment it is necessary to start the Server Manager with: 

start-server-manager

The Server Manager takes care of starting and managing the correct simulation/s and Robot Server/s. Depending on the setup that you choose, the Server Manager could be running on the same machine on which you call env.make() or on another machine connected via network.

The Server Manager is part of the robo-gym-server-modules package. A list of commands is available here.

To start an environment use:

import import gymnasium as gym, robo_gym

env = gym.make('EnvironmentNameSim-v0', ip='<server_manager_address>')
env.reset()

The IP address of the machine on which the Server Manager is running has to be passed as an argument to env.make, if the Server Manager is running on the same machine use ip='127.0.0.1'.

To start a simulated environment with GUI use the optional gui argument:

env = gym.make('EnvironmentNameSim-v0', ip='<server_manager_address>', gui=True)

Additional commands for Simulated Environments

The Simulation wrapper provides some extra functionalities to the Simulated Environments.

Exception Handling Wrapper

The Exception Handling Wrapper comes in handy when training on simulated environments. The wrapper implements reaction strategies to common exceptions raised during training. If one of the know exceptions is raised it tries to restart the Robot Server and the Simulation to recover the system. If the exceptions happen during the reset of the environment the Robot Server is simply restarted in the background, whereas, if exceptions happen during the execution of an environment step the environment returns:

return self.env.observation_space.sample(), 0, True, {"Exception":True, "ExceptionType": <Exception_type>}

Adding the wrapper to any simulated environment is very easy:

import gymnasium as gym
import robo_gym
from robo_gym.wrappers.exception_handling import ExceptionHandling

env = gym.make('EnvironmentNameSim-v0', ip='<server_manager_address>')
env = ExceptionHandling(env)

Real Robot Environments

When making a real robot environment the Robot Server needs to be started manually, see here how to do that.

Once the Real Robot Server is running, you can start the corresponding environment with: 

import gymnasium as gym
import robo_gym

env = gym.make('EnvironmentNameRob-v0', rs_address='<robot_server_address>')

env.reset()

The <robot_server_address> has to be formed as IP:PORT

Environments

back to top

See List of Environments.

For information on creating your own environments, see Creating your own Environments.

Examples

back to top

Random Agent MiR100 Simulation Environment

import gymnasium as gym
import robo_gym
from robo_gym.wrappers.exception_handling import ExceptionHandling

target_machine_ip = '127.0.0.1' # or other machine 'xxx.xxx.xxx.xxx'

# initialize environment
env = gym.make('NoObstacleNavigationMir100Sim-v0', ip=target_machine_ip, gui=True)
env = ExceptionHandling(env)

num_episodes = 10

for episode in range(num_episodes):
    done = False
    env.reset()
    while not done:
        # random step in the environment
        state, reward, done, info = env.step(env.action_space.sample())

Additional examples can be found here

Testing

back to top

Start the Server Manager and attach to the session with: 

start-server-manager && attach-to-server-manager
Expected output

```sh 2021-XX-XX XX:XX:XX,XXX - serverManager - INFO - Server Manager started at 50100 ``` For problems at this step see the [Testing](https://github.com/jr-robotics/robo-gym-server-modules#testing) section of [robo-gym-server-modules](https://github.com/jr-robotics/robo-gym-server-modules).


On the PC where you are running robo-gym associate the IP of the pc on which the Server Manager is running to the hostname robot-servers with:

sudo sh -c 'printf "127.0.0.1 robot-servers" >> /etc/hosts'

If you are running the Server Manager on a different PC replace 127.0.0.1 with the IP address of the machine.

We are using pytest for tests. You can run a short selection of tests with:

pytest -m "not nightly"

or the full test suite with:

pytest

Once you are done run kill-server-manager to kill the Robot Server and the Server Manager.

Troubleshooting

If you encounter troubles running robo-gym please take a look at the existing issues, if you still cannot find solution to your problem please submit a new issue.

Troubleshooting robo-gym-robot-servers Troubleshooting robo-gym-server-modules

Contributing

back to top

New environments and new robots and sensors implementations are welcome!

More details and guides on how to contribute will be added soon!

If you have general questions or ideas that you would like to share please start a new discussion.

Citation

back to top

@article{lucchi2020robo,
  title={robo-gym--An Open Source Toolkit for Distributed Deep Reinforcement Learning on Real and Simulated Robots},
  author={Lucchi, Matteo and Zindler, Friedemann and M{\"u}hlbacher-Karrer, Stephan and Pichler, Horst},
  journal={2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2020}
}

News

back to top