search

moveit / moveit

:robot: The MoveIt motion planning framework
http://moveit.ros.org
BSD 3-Clause "New" or "Revised" License
1.64k stars 943 forks source link

Grasp pose detection (GPD) and Dex-Net support #2188

Open bostoncleek opened 4 years ago

bostoncleek commented 4 years ago

This is an internship project proposed by @mamoll and @JStech. I would appreciate feedback from the community.

Is your feature request related to a problem? Please describe. I want to add more robust grasping support. Currently, moveit grasps has grasp generators for rectangular and cylindrical objects. GPD and Dex-Net are able to generate grasp candidates for objects with a variety of shapes in cluttered scenes using deep learning. GPD generates grasp candidates from a point cloud for a 2 finger gripper. Dex-Net can also generate candidates for a suction gripper.

Describe the solution you'd like I am starting with GPD but hope to move onto Dex-Net if there is time. GPD could be an alternative to the MTC GenerateGraspPose stage.

Describe alternatives you've considered Integrate GPD into moveit grasps or create a separate grasping package. An issue with adding GPD into moveit grasps is that currently GPD requires additional dependencies: Point Cloud Library 1.9 or newer and OpenCV 3.4 or newer. GPD supports Caffe and OpenViINO if you want to train on your own data (but are not required for using GPD).

Additional context The additional dependencies GPD requires can make current packages more cumbersome to get up and running. I have also been working with GPD in simulation using the UR5.

This feature was raised previously #566

Bellow are some grasp candidates generated by GPD using Realsense camera.

welcome[bot] commented 4 years ago

Thanks for reporting an issue. Because we're a volunteer community, providing a pull request with suggested changes is always welcomed.

v4hn commented 4 years ago

The main question I have is what you actually want to do. As I said before, I consider this issue resolved. There are examples that integrate grasps from GPD with MoveIt's grasping pipeline. GPD, as well as spin-offs such as PointNetGPD, publish grasp candidates the whole time and MoveIt-based nodes can simply subscribe there.

Open things in this area are basically

Including any of these grasping frameworks directly in MoveIt is out of the question, you gave some reasons for that yourself.

bostoncleek commented 4 years ago

After discussing it with @mamoll I would like to implement a generator stage for MTC using GPD. We can move this discussion to MTC.