How to get the 3D location of 2D Pixel in reconstructed Mesh?

[darwinharianto]

Checklist

• [X] I have searched for similar issues.
• [X] For Python issues, I have tested with the latest development wheel.
• [X] I have checked the release documentation and the latest documentation (for master branch).

My Question

I reconstructed a 3D Mesh from RGBD image using this function

def generate_mesh(image:np.ndarray, depth_image:np.ndarray, quality):
    height, width, _ = image.shape

    depth_image = (depth_image * 255 / np.max(depth_image)).astype('uint8')
    image = (image * 255 / np.max(image)).astype('uint8')
    # image = np.array(image)

    # create rgbd image
    depth_o3d = o3d.geometry.Image(depth_image)
    image_o3d = o3d.geometry.Image(image)
    rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(image_o3d, depth_o3d,
                                                                    convert_rgb_to_intensity=False)

    # camera settings
    camera_intrinsic = o3d.camera.PinholeCameraIntrinsic()
    camera_intrinsic.set_intrinsics(width, height, 500, 500, width / 2, height / 2)

    # create point cloud
    pcd = o3d.geometry.PointCloud.create_from_rgbd_image(rgbd_image, camera_intrinsic)

    # outliers removal
    cl, ind = pcd.remove_statistical_outlier(nb_neighbors=20, std_ratio=20.0)
    pcd = pcd.select_by_index(ind)

    # estimate normals
    pcd.estimate_normals()
    pcd.orient_normals_to_align_with_direction(orientation_reference=(0., 0., -1.))

    # surface reconstruction
    mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_poisson(pcd, depth=quality, n_threads=1)[0]

    # rotate the mesh
    rotation = mesh.get_rotation_matrix_from_xyz((np.pi, np.pi, 0))
    mesh.rotate(rotation, center=(0, 0, 0))

    # # save the mesh
    # temp_name = next(tempfile._get_candidate_names()) + '.obj'
    # o3d.io.write_triangle_mesh(temp_name, mesh)

    return mesh

I want to segment a part of the image, and get the corresponding part from the mesh. I came up with creating an image with 0 as value for all the part that I don't segment and non-zero for the part that I segmented

Something like this

I want to know where is those white-ish part exist in 3D mesh.

I thought I can get them by comparing the vertex color value, with the color in 2D image, but it seems like vertex_colors is being interpolated. I found some value looking like this (12,13,14) or (253,254,252). Even though my segmented image shoud have the same value for all the channel...

How do I get the location of those segmented parts in 3D coordinates?

[hernot]

How did you do this, do have any knowledge about correspondence between pixel in image and 3d point? Can you retain this information beyond mesh generation/triangulation step? If you simply can use opencv to generate a binary mask separating the interior from the interior of or segmenting polygone. The corresponding surface patch than contains only triangles or tetrahedrons which include only points corresponding to pixels in your 2D enabled by the mask.

[darwinharianto]

How did you do this, do have any knowledge about correspondence between pixel in image and 3d point? Can you retain this information beyond mesh generation/triangulation step? isn't this method convert my rgbd image to 3d point?

o3d.geometry.PointCloud.create_from_rgbd_image # generate point cloud
o3d.geometry.TriangleMesh.create_from_point_cloud_poisson # generate mesh

In the first method, instead of inputting the real rgb image, i put my mask image. So the resulting mesh would have my mask color.

My end result would be a mesh with color that corresponds to my mask. So maybe the this RGB image with corresponding depth image. I am hoping I could get the same exact color (not interpolated) on the generated mesh.

The corresponding surface patch than contains only triangles or tetrahedrons which include only points corresponding to pixels in your 2D enabled by the mask.

Do you mean I should run that mesh generation on each patch mask? So If I have 10 mask instances, I need to run mesh generation 10 times? But If I use the segmented part only, what would the other non segmented RGBD values?

[hernot]

Badly expressed my self. My suggestion was to

a possible approach would be

   1) generate global mesh and texture uvs coordinates for each vertex from rgbd image
   2) segment rgbd image 
   3) generate binary mask representing selected segment
   4) pick first vertex from global mesh
   5) get corresponding uvs rgbd texture coordinates of vertex
   6) test if it falls within the mask
   7) if record its index
   8) pick next vertex from global mesh
   9) if any left startover at 5
   10) done as soon as all vertices from global mesh have been tested

6) select from global mesh only those triangles and corresponding vertices which are formed by vertices linked to an uvs coordinate within your mask only. use to extract the corresponding sub mesh from the global mesh using either TriangleMesh.select_triangles_by_index. use the vertex indices identified in 6

So instead of running a full mesh generation over and over again, use the vertex uvs information ususally used to map pixels/colors within a texture image onto the surface of your objects to map the segmentation of the rgbd image onto the 3D mesh.

The remaining question is whether the methods to generate a 3D mesh from an rgbd image also allow to in addition to extract the uvs coordinates corresponding to each vertex in the mesh.

[darwinharianto]

If I understand it correctly, I would have to compare the mask image with the uv coordinate.

There is a method to calculate uv texture, but I dont understand how can we do number 4-5-6. The uvs would be representing the whole mesh right? I think I have to check if the resulting uv map fit perfectly against the RGB image, if it is, then I can compare it to the mask?

[hernot]

The uvs mesh likely is normalized to image width and height. Thus uvs (0.0,0.0) is top left of image and uvs (1.0,1.0) bottom right of the image. Thus you will likely have to use following formula to convert back and forth between uvs and pixels coordinates.

u = x_{pix} / X_{img}
v = y_{pix} / Y_{img}
x_{pix} = int( u * X_{img} +0.5 ) # is equal to  round( u * x_{img})
y_{pix} = int( v * Y_{img} +0.5 ) # is equal to  round( v * y_{img})

x_{pix} ... x coordinate in pixels relative to image
y_{pix} ... y coordinate in pixels relative to image
X_{img} ... width of image in pixels
Y_{img} ... height of image in pixels 

As you can see in the conversion i used rounding to get x y coordinates closest to actual u and v. You might also choose different approach to get pixel closest to point.

Also be prepared that u<0, v<0, u > 1 and v>1are valid coordinates. You have to decide how you handle them.

Concerning the uvs mesh it should either be a normal TriangleMesh for which has_triangle_uvs() reports true. In that case the triangle_uvs attribute contains len(triangles)*3 uvs points normalized to 1. Each triplet of uvs points in this list corresponds to the triangle idx_{triangle} = int( idx_{uvs} / 3). And the index at which corner of the triangle it is located is tri_{vert} = idx_uvs - idx_{triangle} * 3. So to extract the unique uvs points corresponding to your vertices the easiest is to to convert the triangle vector into a flat array of the same lenght as triangle_uvs vector and than picking form there the unique vertex indices. In python you can use the following

flat_triangles = np.asarray(mesh.triangles).flatten()
valid_vertices,uvs_match = np.unique(flat_triangles,return_index=True)
# uvs_match yields for each valid_vertex the corresponding index into triangle_uvs
# assuming that mesh only contains valid vertices beeing part of at least one triangle 
vertex_uvs = np.asarray(mesh.triangle_uvs)[uvs_match]

picture = np.asarray(image)
picture_vertex_uvs = ( vertex_uvs * [picture.shape[1] ,picture.shape[0]] + 0.5).astype(int)

In case the uvs mesh is just the uvs coordinates I guess you would have to convert it or a copy of it into a full mesh first.

[Gratao]

I have the same problem. I have the x and y coordinates of the 2d image but I haven't found the correspondence to crop only the part that interests me.

Can you help me?

[hernot]

You mean you do not have the triangle uvs stored along with our non textured 3D mesh? eg. cause the image and our mesh were recorded independently by different cameras?