Reinforcement Learning Implementation on a Quadruped
My project in Korea University.
Overview
This project is a reinforment learning environment for a quadruped. It uses python 2.7, ROS (robot operating system) Kinetic Kame, Tensorflow, Gazebo simulation. The robot interacts with both Gazebo simulation and the real world using ROS. It is trained to walk forward in Gazebo simulation and can be deployed in a real robot. Deep Deterministic Policy Gradient(DDPG) is used for the robot.
Files
quadruped folder
This folder contains files for reinforment learning environment.
- config/quadruped_control.yaml : joint controller config for Gazebo
- launch/quadruped_control.launch : Spawn quadruped model with controller in Gazebo
- launch/quadruped_gazebo.launch : Spawn quadruped model only in Gazebo
- launch/rvis_display.launch : display quadruped model in Rvis
- rvis/urdf.rvis : rvis config
- src/model/ : tensorflow model files of learned agent
- src/agent_mem.p : agent Replay buffer memory
- src/eps_rewards.npy : agent episode rewards
- src/ddpg.py : Deep Deterministic Policy Gradient agent written in tensorflow
- src/quadruped_env.py : quadruped environment for gazebo simulation
- src/quadruped_learn.py : learn a ddpg agent
- src/quadruped_gazebo_act.py : learned agent takes actions in Gazebo
- src/quadruped_model.py : quadruped environment for real world (no learning, only actions)
- src/quadruped_act.py : learned agent takes actions in a real quadruped
- src/plot_eps_rewards.py : plot episode rewards from eps_rewards.npy
- urdf/materials.xacro : colors etc. for quadruped urdf
- urdf/quadruped_model.xacro : xacro file for quadruped urdf
- worlds/quadruped.world : Gazebo world file
- CMakeLists.txt : ROS CMakeLists file
- package.xml : ROS package file
quadruped_imu_and_servo folder
This folder contains files for a real robot implementation. ROS should be installed on the real robot. (Running confirmed on Raspberry pi model B + Ubuntu MATE)
quadruped_imu_publisher
- launch/quadruped_imu_publisher_launch.launch : publish IMU readings from adafruit BNO055 9DOF IMU
- src/quadruped_imu_publisher.py : publisher src file
- CMakeLists.txt : ROS CMakeLists file
- package.xml : ROS package file
quadruped_servo_subscriber
- launch/quadruped_servo_subscriber_launch.launch : control servo position(PWM) from published servo position messages
- src/quadruped_servo_subscriber.py : subscriber src file
- CMakeLists.txt : ROS CMakeLists file
- package.xml : ROS package file
Prerequisites
- python 2.7
- ROS Kinetic Kame
- Tensorflow >= 1.4
- Gazebo simulation
additional installations for simulation
-
ROS control
sudo apt-get install ros-kinetic-ros-control
sudo apt-get install ros-kinetic-ros-controllers
sudo apt-get install ros-kinetic-gazebo-ros-control
-
Gazebo simulation update -> update to 7.x (7.0 causes error)
additional installations for real robot
-
How to
- config ROS packages (quadruped, quadruped_imu_publisher, quadruped_servo_subscriber)
Learning an agent
roslaunch quadruped quadruped_control.launch
python quadruped_learn.py
simulate learned agent
roslaunch quadruped quadruped_control.launch
python quadruped_gazebo_act.py
learned agent on real quadruped
-
setup networking for ROS if needed (run roscore on master)
-
on quadruped
roslaunch quadruped_imu_publisher quadruped_imu_publisher_launch.launch
roslaunch quadruped_servo_subscriber quadruped_servo_subscriber_launch.launch
-
on computer running tensorflow
python quadruped_act.py
Tips
always
source ~/catkin_ws/devel/setup.bash
ROS core
roscore
ROS via network (local LAN)
-
MASTER Setting -> get master_ip_addr via ifconfig (master runs roscore)
export ROS_MASTER_URI=http://master_ip_addr:11311
export ROS_IP=master_ip_addr
export ROS_MASTER_URI=http://192.168.0.5:11311
-
SECOND Setting (from other computer)
ssh second_ip_addr -p(port_num)
export ROS_MASTER_URI=http://master_ip_addr:11311
export ROS_IP=second_ip_addr
-
exiting ssh
exit
xacro to URDF file
rosrun xacro xacro --inorder quadruped_model.xacro > model1.urdf
Videos