Standart ROS setup (Code has been tested with ROS-kinetic on Ubuntu 16.04)
Install additional packages
apt-get update && apt-get install -y \
libqt4-dev \
libopencv-dev \
liblua5.2-dev \
virtualenv \
screen \
python3-dev \
ros-kinetic-tf2-geometry-msgs \
ros-kinetic-navigation \
ros-kinetic-rviz
Setup repository:
cd <path_to_catkin_ws>/src/drl_local_planner_ros_stable_baselines
cp .rosinstall ../
cd ..
rosws update
cd <path_to_catkin_ws>
catkin_make -DCMAKE_BUILD_TYPE=Release
(please install missing packages)
Setup virtual environment to be able to use python3 with ros (consider also requirements.txt)
virtualenv <path_to_venv>/venv_p3 --python=python3
source <path_to_venv>/venv_p3/bin/activate
<path_to_venv>/venv_p3/bin/pip install \
pyyaml \
rospkg \
catkin_pkg \
exception \
numpy \
tensorflow=="1.13.1" \
gym \
pyquaternion \
mpi4py \
matplotlib
cd <path_to_catkin_ws>/src/drl_local_planner_forks/stable_baselines/
<path_to_venv>/venv_p3/bin/pip install -e path_to_catkin_ws>/src/drl_local_planner_forks/stable-baselines/
Set system-relevant variables
Train agent
roscore
roslaunch rl_bringup setup.launch ns:="sim1" rl_params:="rl_params_scan"
source <path_to_venv>/bin/activate
python rl_agent/scripts/train_scripts/train_ppo.py
roslaunch rl_bringup rviz.launch ns:="sim1"
Execute self-trained ppo-agent
roscore
roslaunch rl_bringup setup.launch ns:="sim1" rl_params:="rl_params_scan"
source <path_to_venv>/venv_p3/bin/activate
roslaunch rl_agent run_ppo_agent.launch mode:="train"
roslaunch rl_bringup rviz.launch ns:="sim1"
Note: To be able to load the pretrained agents, you need to install numpy version 1.17.0.
<path_to_venv>/venv_p3/bin/pip install numpy==1.17
roscore
roslaunch rl_bringup rviz.launch ns:="sim1"
roslaunch rl_bringup setup.launch ns:="sim1" rl_params:="rl_params_scan"
source <path_to_venv>/venv_p3/bin/activate
roslaunch rl_agent run_1_raw_disc.launch mode:="train"
source <path_to_venv>/venv_p3/bin/activate
roslaunch rl_agent run_3_raw_disc.launch mode:="train"
source <path_to_venv>/venv_p3/bin/activate
roslaunch rl_agent run_1_raw_cont.launch mode:="train"
roslaunch rl_bringup setup.launch ns:="sim1" rl_params:="rl_params_img"
source <path_to_venv>/venv_p3/bin/activate
roslaunch rl_agent run_1_img_disc.launch mode:="train"
I set up a docker image, that allows you to train a DRL-agent in parallel simulation environments. Furthermore, it simplifies the deployment on a server. Using docker you don't need to follow the steps in the Installation section.
Build the Docker image (This will unfortunately take about 15 minutes):
cd drl_local_planner_ros_stable_baselines/docker
docker build -t ros-drl_local_planner .
In start_scripts/training_params/ppo2_params, define the agents training parameters.
Parameter | Desctiption |
---|---|
agent_name | Number of timestamps the agent will be trained. |
total_timesteps | Number of timestamps the agent will be trained. |
policy | see PPO2 Doc |
gamma | see PPO2 Doc |
n_steps | see PPO2 Doc |
ent_coef | see PPO2 Doc |
learning_rate | see PPO2 Doc |
vf_coef | see PPO2 Doc |
max_grad_norm | see PPO2 Doc |
lam | see PPO2 Doc |
nminibatches | see PPO2 Doc |
noptepochs | see PPO2 Doc |
cliprange | see PPO2 Doc |
robot_radius | The radius if the robot footprint |
rew_func | The reward functions that should be used. They can be found and defined in rl_agent/src/rl_agent/env_utils/reward_container.py. |
num_stacks | State representation includes the current observation and (num_stacks - 1) previous observation. |
stack_offset | The number of timestamps between each stacked observation. |
disc_action_space | 0, if continuous action space. 1, if discrete action space. |
normalize | 0, if input should not be normalized. 1, if input should be normalized. |
stage | stage number of your training. It is supposed to be 0, if you train for the first time. If it is > 0, it loads the agent of the "pretrained_model_path" and continues training. |
pretrained_model_name | If stage > 0 this agent will be loaded and training can be continued. |
task_mode | - "ped" for training on pedestrians only; "static" for training on static objects only; "ped_static" for training on both, static |
There are some predefined agents. As example I will use the ppo2_1_raw_data_disc_0 in the training session.
docker run --rm -d \
-v <folder_to_save_data>:/data \
-v drl_local_planner_ros_stable_baselines/start_scripts/training_params:/usr/catkin_ws/src/drl_local_planner_ros_stable_baselines/start_scripts/training_params \
-e AGENT_NAME=ppo2_1_raw_data_disc_0 \
-e NUM_SIM_ENVS=4 \
ros-drl_local_planner
If you want to display the training in Rviz, run the docker container in the hosts network. In order to use rviz, the relevant packages need to be compiled on your machine.
docker run --rm -d \
-v <folder_to_save_data>:/data \
-v drl_local_planner_ros_stable_baselines/start_scripts/training_params:/usr/catkin_ws/src/drl_local_planner_ros_stable_baselines/start_scripts/training_params \
-e AGENT_NAME=ppo2_1_raw_data_disc_0 \
-e NUM_SIM_ENVS=4 \
--net=host \
ros-drl_local_planner
Now you can display the different simulation environments:
roslaunch rl_bringup rviz.launch ns:="sim1"
roslaunch rl_bringup rviz.launch ns:="sim2"
```
docker run --rm -d \
-v drl_local_planner_ros_stable_baselines/example_agents:/data/agents \
-v drl_local_planner_ros_stable_baselines/start_scripts/training_params:/usr/catkin_ws/src/drl_local_planner_ros_stable_baselines/start_scripts/training_params \
-e AGENT_NAME=ppo2_1_raw_data_disc_0_pretrained \
-e NUM_SIM_ENVS=4 \
--net=host \
ros-drl_local_planner
```
```
docker run --rm -d \
-v drl_local_planner_ros_stable_baselines/example_agents:/data/agents \
-v drl_local_planner_ros_stable_baselines/start_scripts/training_params:/usr/catkin_ws/src/drl_local_planner_ros_stable_baselines/start_scripts/training_params \
-e AGENT_NAME=ppo2_1_img_disc_1_pretrained \
-e NUM_SIM_ENVS=4 \
--net=host \
ros-drl_local_planner
```