RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.19k stars 1.24k forks source link

DepthImageToPointCloud does not handle frame offsets #12125

Open RussTedrake opened 4 years ago

RussTedrake commented 4 years ago

@pangtao22 noticed that the RGB ad point clouds are off for the physical robot, and realized that our DepthImageToPointCloud class does not handle the case when the RGB camera frame is offset from the depth camera frame.

Image from iOS

I'm not actually sure what the "right" algorithm is here (the projection seems nontrivial?). to leave myself a breadcrumb, the open3d version is here: http://www.open3d.org/docs/release/python_api/open3d.geometry.RGBDImage.html#open3d.geometry.RGBDImage.create_from_color_and_depth but opens with "RGBDImage is for a pair of registered color and depth images, viewed from the same view, of the same resolution. If you have other format, convert it first."

EricCousineau-TRI commented 3 years ago

From reviewing #14985, I believe the defect is as you mentioned; the depth image is not registered to same camera extrinsics nor intrinsics, due to this code snippet: https://github.com/RobotLocomotion/drake/blob/85525640efa7f40bb2f7abad6700d70e74c0d6b0/examples/manipulation_station/manipulation_station.cc#L167-L186

Possible solutions:

\cc @SeanCurtis-TRI