Do we need to compensate for the 3d box annotation when we project it to the image?

[xmyqsh]

I noticed that when we project lidar to the image, we should compensate for the motion due to the unperfect time sync which is done in project_lidar_to_img_motion_compensated.

I think we should also need to compensate for the 3d box annotation when we project it to the image. Because the annotation timestamp is same as the lidar sweep. But project_ego_to_cam does not compensate it.

[johnwlambert]

Hello @xmyqsh, thanks for your interest in our work.

As you noted, we provide a function project_lidar_to_img_motion_compensated() to project LiDAR points into an image (and compensating for the egomotion between the two sensor observations) here. The annotations are present in the LiDAR sweeps, so we only need to consider compensating between LiDAR and images.

This function requires providing the timestamp of both sensor observations:

                    if motion_compensate:
                        uv, uv_cam, valid_pts_bool, K = project_lidar_to_img_motion_compensated(
                            points_h,  # these are recorded at lidar_time
                            copy.deepcopy(log_calib_data),
                            camera_name,
                            cam_timestamp,
                            lidar_timestamp,
                            data_dir,
                            log_id,
                            return_K=True,
                        )

The function works by moving LiDAR points in the egovehicle frame at time t0 into the egovehicle frame at time t1 when the image was recorded, using the map as a static reference frame to unify the two.

    ego_cam_t_SE3_ego_lidar_t = city_SE3_ego_cam_t.inverse().right_multiply_with_se3(city_SE3_ego_lidar_t)
    pts_h_cam_time = ego_cam_t_SE3_ego_lidar_t.transform_point_cloud(pts_h_lidar_time)

As you noted, we also provide a version of such a projection function without any motion compensation, simply for the sake of illustration and completion (here). I would not recommend using project_ego_to_cam (and we do not use it in our demo file's default settings either). Without motion compensation, image projection will look very choppy and not nearly as smooth or accurate.

[xmyqsh]

@johnwlambert Thanks for your reply. ~~As you said that the annotations are present in the LiDAR sweeps, I think it is also necessary to consider compensating between 3d annotations and images. Right?~~

But it seems that some of the annotations are not very accurate, no matter I compensated it or not.

root_dir =  '../argoverse-tracking/train1/'
log_id = 'c6911883-1843-3727-8eaa-41dc8cda8993'
idx = 0

---

While, when idx = 150 it looks very well:

Can you help to have a double-check?

[johnwlambert]

Hi @xmyqsh, I took a look at the video and it looks like either the LiDAR or the camera starts recording momentarily before the other, so the LiDAR annotation seems to stand still for the first, second, and third image, before we start seeing full sensor observations kick in. This leads to a misalignment of ~5-10 cm (not noticeable on a car or person, but only on a bollard) for the first two images.

The LiDAR observation occurs once per 100 ms and the image observation occurs once per ~33 ms.

As far as I can tell, the LiDAR-camera alignment/synchronization is correct for every single other object (hundreds of other bounding boxes) in the video except for those two skinny bollards in the first frame.

I would recommend discarding the first LiDAR frame of this sequence if these bollards present an issue for you.

I've uploaded the relevant videos here: https://youtu.be/XlRpSpFL5ZI

https://youtu.be/Qnb2AgDHozI