How to convert range image based lidar semantic labels to 3D point cloud label format in v2 dataset

[SM1991CODES]

Hi, Please advise how can we take the base segmentation labels available as LaserSegmentationRangeImage and convert to 3D point cloud format to be used with 3D point cloud derived from the lidar range images. I cannot find any tutorial for this.

In fact, there is extremely small amount of information about working with v2 dataset.

[SM1991CODES]

Is this project active anymore? Could someone please advise?

[JingweiJ]

Hi @SM1991CODES ,

The V2 dataset primarily features a different data format optimized for modular reading. Once loaded, you can process the data similarly to how you would with the V1 dataset. You may try this function to convert the segmentation label range image to point cloud:

def convert_range_image_to_point_cloud_labels_v2(
    range_image: v2.perception.lidar.RangeImage,
    segmentation_label: v2.perception.segmentation.LiDARSegmentationRangeImage
) -> np.ndarray:
  range_image_tensor = range_image.tensor
  range_image_mask = range_image_tensor[..., 0] > 0
  sl_tensor = segmentation_label.tensor
  sl_points_tensor = tf.gather_nd(sl_tensor, tf.where(range_image_mask))
  return sl_points_tensor.numpy()

This is similar to the convert_range_image_to_point_cloud_labels function in tutorial/tutorial_3d_semseg.ipynb.

Be sure that you select the (LiDAR) range image and segmentation label that correspond with the matched laser name and frame timestamp.

[SM1991CODES]

Hello, Thank you for the help - this basically works. But. I have another problem - I am trying to use v2.merge to combine the fillowing components: lidar_df = read(path_dataset_root, components["lidar"], context_name) lidar_labels_df = read(path_dataset_root, components["lidar_labels"], context_name) lidar_calibs_df = read(path_dataset_root, components["lidar_calibs"], context_name) ego_pose_df = read(path_dataset_root, components["ego_pose"], context_name) lidar_seg_labels_df = read(path_dataset_root, components["lidar_seg"], context_name)

    lidar_calibs_df = v2.merge(lidar_df, lidar_calibs_df)
    lidar_calibs_pose_df = v2.merge(lidar_calibs_df, ego_pose_df)

    lidar_labels_sem_df = v2.merge(lidar_seg_labels_df, lidar_labels_df)
    lidar_calibs_labels_df = v2.merge(lidar_calibs_pose_df, lidar_labels_sem_df, left_group=True, right_group=True)

I am trying to get for each frame - point cloud, object labels, ego pose, semantic labels This does not work. I am aware that waymo does not provide semantic labels for each frame.

Please advise how to do this correctly.

[JingweiJ]

Hi @SM1991CODES , have you tried to filter the DFs to keep only rows of the frames with semantic labels?