Create Point Cloud from depth image.

[RitchieQi]

Hi @samarth-robo:

I tried to create point cloud from depth image to match the 3D joint coordinates that got from _cTo@oX (which has been discussed in #23), but here's a problem about the creation of Point Cloud. Following is my code:

point_image = o3dio.read_image(depth_dir)
intrinsics = o3dc.PinholeCameraIntrinsic()
intrinsics.intrinsic_matrix = np.dot(cp.A(camera_name), cp.K(camera_name))
extrinsics = cp.object_pose(camera_name,frame_idx)
pc = o3dg.PointCloud.create_from_depth_image(depth=point_image, intrinsic = intrinsics, extrinsic = extrinsics,depth_scale = 1000.0)
print(np.asarray(pc.points))
o3dv.draw_geometries([pc])

The output in terminal is:

[[nan nan nan]
 [nan nan nan]
 [nan nan nan]
 ...
 [nan nan nan]
 [nan nan nan]
 [nan nan nan]]

And nothing is showed on the Open3d window. Do you have any idea about what is the reason for this?

Thanks in advance!

[RitchieQi]

After doing some investigation and experiments, I get the point cloud aligned with the 3D joints coordinates. Here I put my thoughts and solution:

1. The transformation tree

World frame | |----wTc----Camera frame----intrinsics----Pixel frame----affine----ContactPose |        | |        cTo |        | |----wTo----Object frame

2. The 3D joint coordinates

In #23, I transformed the joints coordinates from object frame to camera frame by

cTo @ Ho = Hc

The cTo, a.k.a ContactPose.object_pose(camera_name,frame_idx), and Ho, a.k.a ContactPose.hand_joints(frame_idx), both could be accessed directly from ContactPose class.

3. The point cloud

According to @samarth-robo, the original images have been flipped and/or transposed. More information could be found here https://github.com/facebookresearch/ContactPose/issues/10#issuecomment-817374363. But instead of using ContactPose.A@ContactPose.K as intrinsic matrix in creating point cloud, I think it should be done individually, which is, inverse the affine transformation on depth image first, then create point cloud with K as the intrinsic matrix. This is because the affine transformation matrix is dedicated for 2D coordinate and would cause problem when use it in 3D point cloud creation. The details about affine transformation could be found here ContactPose.get_A. It could be inversed by applying numpy.flipud, numpy.fliprl and numpy.transpose to the images.

The next step is converting the original depth image into point cloud in camera frame:

pcd = o3dg.PointCloud.create_from_depth_image(depth=point_image, intrinsic = ContactPose.K(camera_name),  depth_scale = 1000.0)

Since I have already transform the joints coordinates into camera frame, I don't need further extrinsic transformation to convert point cloud to the object frame. Here is the result (have adjusted the viewpoint for better visualisation).

[samarth-robo]

Thanks a lot for the detailed post, @RitchieQi! I am curious, did the other way mentioned in https://github.com/facebookresearch/ContactPose/issues/10#issuecomment-817374363 using K' = A @ K not work properly? If so, I should document that somewhere.

[RitchieQi]

Anytime, @samarth-robo. Actually no, at least it didn't work for me as I described in the first comment of this issue https://github.com/facebookresearch/ContactPose/issues/24#issue-1239733257.