panda-deep-grasping

Linux Ubuntu Focal (20.04) | ROS Noetic

Installation

Simulation and evaluation are using different verisons of franka_ros and libfranka, so you need two separate workspaces. The simulation workspace requires some extra steps. The location to save and load the replay buffer files, model files and training or evaluation data has to be set manually.

Step 1: Installation for both Evaluation and Simulation

1. Install ROS Noetic
2. Create two ROS-Workspaces. One for the simulated environment, one for the real environment. Recommended file structure and workspace names from user folder:

   panda_deep_grasping
   |
   +--- simulation_ws
   |
   +--- real_ws

3. Install MoveIt from packages
   sudo apt install ros-noetic-moveit ros-noetic-moveit-visual-tools
4. Install Gazebo from packages
5. Install gazebo_ros_packages from packages
   sudo apt-get install ros-noetic-gazebo-ros-pkgs ros-noetic-gazebo-ros-control
6. Install OpenCV from packages
   sudo apt install libopencv-dev python3-opencv
7. Install realsense_ros
   sudo apt-get install ros-noetic-realsense2-camera
8. Install franka_ros from packages
   sudo apt install ros-noetic-libfranka ros-noetic-franka-ros
9. Install tensorflow
   pip install tensorflow
10. Install panda_deep_grasping package by cloning in simulation_ws/src and real_ws/src:
    git clone https://git.scc.kit.edu/uhksn/panda-deep-grasping.git

Step 2: Additional steps for Simulation

Following additional steps are necessary for the simulation workspace:

1. Install panda_moveit_config from source: In simulation_ws/src:
   git clone -b noetic-devel https://github.com/vincentmllr/panda_moveit_config.git
2. Install franka_ros 9.0.0 from source: In simulation_ws/src:
   git clone -b 0.9.0 https://github.com/vincentmllr/franka_ros.git
3. Add Camera to urdf description: Exchange panda_gazebo.xacro file from "franka_ros/franka_description/robots/" with the version in "panda_deep_grasping/description/": In simulation_ws/src:
   cd panda-deep-grasping/description && cp panda_gazebo.xacro ../../franka_ros/franka_description/robots/panda_gazebo.xacro && cd ../..

Step 3: Finish Installation

Following commands should be done once after the previous steps:

1. In workspace folders:
   catkin config --extend /opt/ros/${ROS_DISTRO} --cmake-args -DCMAKE_BUILD_TYPE=Release
2. In src folders:
   rosdep install -y --from-paths . --ignore-src --rosdistro noetic
3. In one workspace folder:
   catkin_make
4. In one workspace folder:
   source devel/setup.bash
   Has to be done in every new terminal and after catkin_make. If you are only working with one workspace you can automatically source in every new terminal with:
   echo "source ~/catkin_ws/devel/setup.bash" >> ~/.bashrc

Startup

If you didnt use the recommended file structure, you have to set the file paths for the replay-buffer-files, the model-files and the training-data-file. You can change those filepaths by changing the according variables at the top of the agent_main, agent_model, agent_buffer, environment_sim and environment_real-files. Additionally, using those variables you can choose whether you want to load an existing model or replay buffer or which version name the files should be saved under.

Training in simulation

1. cd ~/panda_deep_grasping/simulation_ws ; source devel/setup.bash
2. roslaunch panda_deep_grasping training.launch

Evaluation in simulation

1. Change ONLY_ACTOR variable in agent_model.py to true
2. cd ~/panda_deep_grasping/simulation_ws ; source devel/setup.bash
3. roslaunch panda_deep_grasping evaluation_sim.launch

Evaluation in real environment

1. Real Robot black button has to be unlocked (not pressed down)
2. Make sure camera is connected via USB
3. Change ONLY_ACTOR variable in agent_model.py to true
4. cd ~/panda_deep_grasping/real_ws ; source devel/setup.bash
5. roslaunch panda_deep_grasping evaluation_real.launch

Structure

panda-deep-grasping
|
+--+ description: 3D models for simulation
|  |
|  +---panda_gazebo
|  +---realsense_d435
|  +---cube
|  +---tabletop
|
+--+ launch: files to startup training or evaluation
|  |
|  +--- evaluation_real.launch
|  +--- evaluation_sim.launch
|  +--- training.launch
|
+--- models: trained neural networks
|
+--+ msg: message description for communication over ROS
|  |
|  +--- Action.msg
|  +--- Reward.msg
|  +--- Sample.msg
|  +--- State.msg
|
+--+ replay_buffer_files: saved training samples and images
|  |
|  +--- replay_buffer.txt
|  +--- replay_images
|   
+--+ results
|  |
|  +--- training_data.txt
|
+--+ scripts
|  |
|  + agent_main.py: handles training loop
|  + agent_model.py: neural networks
|
+--+ src
|  |
|  + environment_real.cpp: interface to real environment
|  + environment_sim.cpp: interface to simulated environment
|  + agent_buffer.cpp: saving and retrieving of training samples
|
+--+ srv: service description for communication over ROS
|  |
|  +---EnvironmentStep.srv
|  +---GetAction.srv
|  +---GetBatch.srv
|  +---ResetEnvironment.srv
|  +---Train.srv
|
+--- CMakeLists.txt
+--- package.xml
+--- README_LOCAL.md
+--- README.md