Downsample point cloud to known resolution (this increases speed and ensures that the algorithm works even if the distance from the camera to the object changes)
Select a set of points right around where the user clicked, this is the starting point for the optimization
Sement the object
Randomly grow the selection (only points within a specified radius of the currently active points are considered)
Calculate the error on the new selection. The objective function is weighted sum of the convexity of the point cloud (the distance between the point cloud and its convex hull) and the number of points
Use simulated annealing to decide whether to keep the new selection or not
Repeat until converged (usually < 10 iterations)
Generate a bounding box
How the bounding box is generated has a large impact on the output grasp
An axis-aligned bounding box is simple and generates straight-on grasps, but may not work well with objects that are at an angle
An automatically generated bounding box (with one of the three axes aligned with the SE2 orientation specified by the user) is more complex, but can work better with objects that are rotated in more than one axis
Pick a face of the bounding box that is facing out of the cell, and generate a 6DoF grasp pose based on it
Remaining questions
Still need to test this on a larger variety of objects, especially mesh / bagged (how is the depth data affected?)
Fully Manual
In a fully manual setup, the user now needs to specify the width in addition to XY position and orientation relative to the camera. There are a few methods of specifying width:
Preset width buttons (similar to speed presets in stretch_web_interface)
Take mouse release position into account for SE2 press/release to get distance
User specifies the location of the both fingers
User specifies the center of the grasp and the width of the fingers
In a fully manual setup I think it also makes the most sense for the approach direction to straight on (unless we add in additional camera views / DoF)
That means that the only unconstrained grasp variable is how far the gripper should go into the bin. I think it is okay to just calculate the location of the point cloud and use that. Its not fully manual, but it should be pretty simple and accurate.
The issue of inaccurate depth data is is an issue here too, though a smaller one
Fully Automatic
Grasp Generation Pipeline
Remaining questions
Fully Manual
In a fully manual setup, the user now needs to specify the width in addition to XY position and orientation relative to the camera. There are a few methods of specifying width:
In a fully manual setup I think it also makes the most sense for the approach direction to straight on (unless we add in additional camera views / DoF) That means that the only unconstrained grasp variable is how far the gripper should go into the bin. I think it is okay to just calculate the location of the point cloud and use that. Its not fully manual, but it should be pretty simple and accurate. The issue of inaccurate depth data is is an issue here too, though a smaller one