GPD (Grasp Pose Detection) Library Integration

[davetcoleman]

Andreas ten Pas (@atenpas) has released an excellent ROS library called GPD for detecting grasp poses in point clouds using the Caffe deep learning framework and PCL. However there is currently no integration with MoveIt!. Myself and @v4hn are collaborating with @atenpas on creating this integration and this issue will serve as a public communication point as well as a call for assistance or collaboration.

Steps to achieve this goal:

• [ ] Create or adapt a ROS node that can communicate with MoveIt! MoveGroup's grasp generation action.
• [ ] Create a UR5 demo using pipeline
• [ ] Write a MoveIt! tutorial on installing dependencies (including CUDA drivers) and running demo
• [ ] Create blog post to announce feature on behalf of the Northeastern Help Hands Lab

@tkelestemur mentioned he has already been using this package and perhaps can advise.

[tkelestemur]

We have been using @atenpas' grasp pose detection code with MoveIt!. We deployed the code on Toyota's HSR robot and got pretty good results. I will try to do my best to share my experience about how to integrate GPD with MoveIt!. Unfortunately the code that I wrote for the HSR is quite a mess (I had to put everything in the last minute for the RoboCup competition).

There are 3 easy steps to integrate GPD to your MoveIt pipeline.

1. Create a perception code that takes raw point cloud from a kinect-like sensor and publish it as on of the GPD package messages. (you need to some processing which is explained in GPD's github repo)
2. Map your robot's end-efffector frame to GPD's default frame.
3. Transform your returned grasp poses to your planning frame (assuming that your perception codes published GPD msg w.r.t. sensor link)

After that you can just send this pose to your planner or IK solver.

One trick I did was that before sending to IK/planner I defined a pre-grasp pose which 10 cm back from the approach vector because if you directly use returned poses, IK will give you collision.

@davetcoleman I have not used grasp generation action before. What you want to do here is that create a node that will act as a pickup action server by using GPD. If so, I think that can be doable easily.

[v4hn]

Here's an excerpt from a mail I wrote on this matter. This focuses on a "good" high-level integration, instead of the details @tkelestemur explained above.

The current workflow is as follows:

• Add the object you wish to grasp to the planning scene. In your packages there is usually no notion of an "object to grasp", right? A reasonable solution that comes to my mind, without adding another object pose estimation module to the pipeline, is to specify some 3d bounding box as collision object and restrict the set of valid grasps to those near the bb, but there are many ways to do this, I guess.

• The client can then call MoveGroupInterface::planGraspsAndPick and provide the collision object or the id of the thing to grasp. https://github.com/TAMS-Group/tams_ur5_pick_place/blob/master/src/pick_and_place_test.cpp#L41

• This is passed on to a ROS service you have to implement - this is the interface in which we have to wrap GPD somehow. The service receives the target collision object, e.g. bounding box, and creates moveit_msgs::Grasp messages from that. https://github.com/TAMS-Group/tams_ur5_pick_place/blob/master/src/plan_grasps_service.cpp#L20 It is clearly possible to have the same node subscribe to point cloud topics to get the input you need.

• The result is automatically forwarded to the grasp pipeline which tries to create a hybrid cartesian/joint space trajectory that adheres to the grasp message and goes on to pick the object if successful.

@tkelestemur Your explanation sounds like you use neither the moveit_msgs/Grasp message nor the pick/place pipeline. Is that right? In this case, I guess you also didn't encounter the problem of the non-existing collision object? Did you think about it though? Without an attached object the planned paths do not consider collisions between the grasped object and the environment.

[davetcoleman]

@tkelestemur can you link us to the code you wrote for GPD, even if its a mess? It might serve as a good starting point for a cleaned up version.

[atenpas]

Add the object you wish to grasp to the planning scene. In your packages there is usually no notion of an "object to grasp", right? A reasonable solution that comes to my mind, without adding another object pose estimation module to the pipeline, is to specify some 3d bounding box as collision object and restrict the set of valid grasps to those near the bb, but there are many ways to do this, I guess.

Do we want bounding boxes or is there something in moveit that can detect objects?

[tkelestemur]

@v4hn My pipeline simply was:

1. Segment table and tabletop objects.
2. Create and publish GPD CloudIndexed msg.
3. Choose the closest grasp pose from top ten grasp poses (which has IK solutions).
4. Plan and move to pre-grasp pose (which is 10cm away in approach vector).
5. Plan and move from pre to actual grasp pose (using Cartesian path planner).
6. Attach a collision object (with predefined size) to the robot.
7. Lift up the object 15cm up.
8. Plan and move to placing pose.

@davetcoleman I created a gist that shows two function that is being used in grasp planning pipeline. You can access from here. Unfortunately I cannot make the whole repo public since it has other research-related codes.

@atenpas There are some packages does tabletop segmentation. I remember two: http://wiki.ros.org/rail_segmentation and https://github.com/mikeferguson/simple_grasping. I wrote my custom segmenter tho.

[davetcoleman]

Do we want bounding boxes or is there something in moveit that can detect objects?

Perception is outside the scope of MoveIt!, except for the octomap integration which is debatable. So no, we don't have the ability to detect objects without third-party software. MoveIt! can accept into its planning scene representation bounding boxes, meshes, or other shape primitives. I presume we want bounding boxes, yes.

[xmba15]

@tkelestemur Did you create a hsr_15_channels.launch when you used gpd on HSR? Can you share details about the launch file to use gpd on HSR?

[v4hn]

My lab worked on GPD integration and the current outcome is https://github.com/TAMS-Group/moveit_gpd_pick_object

This works well with our setup and should be flexible enough to allow for other grippers too.

Personally I consider this issue resolved for now. And further improvements can be discussed in the other repository.

@davetcoleman any objections to closing this issue?

[tkelestemur]

@bajsk I've used default classifier_15channels.launch .

[bringBackm]

Hi all , Is there any progress in this matter?

[v4hn]

What kind of progress to you propose? If you want to use GPD with MoveIt you can have a look at moveit_gpd_pick_object.

If you want to propose further ideas for integration, go ahead please.

[bringBackm]

Ok, thank you , l just wondering whether there are some tutorials about the GPD grasps. It seems that Moveit! tutorials lack the part of grasps and the usage of the Grasp.msg

[v4hn]

It seems that Moveit! tutorials lack the part of grasps and the usage of the Grasp.msg

That's true, documentation lacks. I hope @davetcoleman and @mlautman will improve this with their overhaul of the tutorials :smile:

[bringBackm]

Yeah, looking forward to it!

[davetcoleman]

fyi I'm afraid grasping did not make the cut for this round of tutorial updates

[sharronliu]

Hi all, I've spent some efforts enabling GPD with the Intel OpenVINO toolkit. This toolkit extends the NN computation workloads across Intel hardware (CPU, GPU, Movidius NCS, FPGA, etc.) and maximizes performance.

[sharronliu]

I'm now doing the grasping setup, and maybe I can help to continue updating the tutorials. Can anybody share the linkage to the existing tutorials where I can start from? Thanks.

[felixvd]

You can fork this project and submit a pull request when you are ready.

[qyp-robot]

My lab worked on GPD integration and the current outcome is https://github.com/TAMS-Group/moveit_gpd_pick_object

This works well with our setup and should be flexible enough to allow for other grippers too.

Personally I consider this issue resolved for now. And further improvements can be discussed in the other repository.

@davetcoleman any objections to closing this issue?

Thank you for your answer, it solved my doubts, it is very helpful for ROS beginners!