Annotation Boxes are not compare with the LiDAR point cloud.

[zeng-hello-world]

The 3d boxes are not compare with the LiDAR point cloud as below. The top right are bboxes and the bottom left is point cloud.

[ternaus]

Could you please share more information, so that we would be able to reproduce this bug?

[zeng-hello-world]

I use Open3D to visualize the point cloud and boxes.

level5data = NuScenes(version='v1.01-train', dataroot=DATA_ROOT, verbose=True)

for i, scene in enumerate(level5data.scene):
sample_token = scene['first_sample_token']
    j = 0
    while sample_token != scene['last_sample_token']:
        sample = level5data.get('sample', sample_token)
        # point cloud
        top_lidar_sample_data = level5data.get('sample_data', sample['data']['LIDAR_TOP'])
        top_lidar_file_name = DATA_ROOT + '/' + top_lidar_sample_data['filename']
        if os.path.isfile(top_lidar_file_name):
            pointcloud = LidarPointCloud.from_file(top_lidar_file_name)
        else:
            print("ERROR")

        pcd = open3d.geometry.PointCloud()
        pcd.points = open3d.utility.Vector3dVector(pointcloud.points[:3].transpose())

        # visualization
        mesh_frame = open3d.geometry.TriangleMesh.create_coordinate_frame(size=2.0, origin=[0.0, 0.0, 0.0])
        vis_list = [pcd, mesh_frame]

        # 3D bboxes
        boxes = level5data.get_boxes(sample['data']['LIDAR_TOP'])
        for k, box in enumerate(boxes):
            corner_pts = box.corners().transpose()
            lines = [[0, 1], [1, 2], [2, 3], [3, 0], [4, 5], [5, 6], [6, 7], [7, 4], [0, 4], [1, 5], [2, 6], [3, 7]]
            colors = [[1, 0, 0] for m in range(len(lines))]
            tmp_box_line = open3d.geometry.LineSet()
            tmp_box_line.points = open3d.utility.Vector3dVector(corner_pts)
            tmp_box_line.lines = open3d.utility.Vector2iVector(lines)
            tmp_box_line.colors = open3d.utility.Vector3dVector(colors)
            vis_list.append(tmp_box_line)

        open3d.visualization.draw_geometries(vis_list)
        sample_token = level5data.get("sample", sample_token)["next"]
        j += 1

[ternaus]

Could you please update to a more recent version of the repo and verify that the issue still exists?

[pyaf]

Looks like OP is plotting the lidar point cloud together with the default Box.corners() without moving them to the car's reference frame?

[pyaf]

@ternaus Suppose we want to plot the lidar point cloud together with the annotations present in that point cloud in a 3D plot.

sample = level5data.get('sample', sample_token)
top_lidar_sample_data = level5data.get('sample_data', sample['data']['LIDAR_TOP'])
top_lidar_file_name = DATA_ROOT + '/' + top_lidar_sample_data['filename']
pointcloud = LidarPointCloud.from_file(top_lidar_file_name)
boxes = level5data.get_boxes(sample['data']['LIDAR_TOP'])

At this point we have the lidar point cloud as well as the boxes associated with our LIDAR_TOP sample_data, no we gotta move the box in car's reference frame, right? before plotting them on same 3D plot?

[gzuidhof]

@pyaf you're right: to plot the annotations and pointclouds together you will need to have both in the same reference frame. The annotations are in global frame, and the sensors are in the sensor's reference frame (and there is a third reference frame that is the car reference frame).

In section B of the reference model on Kaggle you can see an example of how to do this (there they are both transformed to car reference frame).