Open RussTedrake opened 4 years ago
From reviewing #14985, I believe the defect is as you mentioned; the depth image is not registered to same camera extrinsics nor intrinsics, due to this code snippet: https://github.com/RobotLocomotion/drake/blob/85525640efa7f40bb2f7abad6700d70e74c0d6b0/examples/manipulation_station/manipulation_station.cc#L167-L186
Possible solutions:
cv2.registerDepth
. The math is not too hard to write in C++; dunno how great performance is.\cc @SeanCurtis-TRI
@pangtao22 noticed that the RGB ad point clouds are off for the physical robot, and realized that our DepthImageToPointCloud class does not handle the case when the RGB camera frame is offset from the depth camera frame.
I'm not actually sure what the "right" algorithm is here (the projection seems nontrivial?). to leave myself a breadcrumb, the open3d version is here: http://www.open3d.org/docs/release/python_api/open3d.geometry.RGBDImage.html#open3d.geometry.RGBDImage.create_from_color_and_depth but opens with "RGBDImage is for a pair of registered color and depth images, viewed from the same view, of the same resolution. If you have other format, convert it first."