Misalignment between camera data frames and LiDAR data frames

[Devoe-97]

We projected the 3D bounding boxes on the LiDAR coordinate system onto the image and found the misalignment problem. LiDAR id: 000009 Camera id: 000009

LiDAR id: 000009 Camera id: 000010

[thaotrongtran]

For 000009 frame If you try using this codebase, the bounding box looks more correct but still has some alignment issues. If you look at consecutive frames, the bounding boxes change significantly, which should not happen.

I checked the alignment of objects in the Camera coordinate system between LiDAR points and bounding boxes and found no issues, which makes me suspect that the issue is actually with the intrinsic matrix.

[Devoe-97]

For 000009 frame If you try using this codebase, the bounding box looks more correct but still has some alignment issues. If you look at consecutive frames, the bounding boxes change significantly, which should not happen.

Perhaps the reason is that the rotation matrix is not really proper, I hope the authors can clarify r_velo2cam [[-0.06380332 -0.99109149 -0.04429948] [-0.21028734 0.04399769 -0.79876929] [ 0.97575115 -0.06031493 -0.17158543]] angles 90.280 89.763 88.653

However, also I check the alignment of objects in the Camera coordinate system between LiDAR and bounding boxes and found no issues, which makes me suspect that the issue is actually with the intrinsic matrix.

Hi, could you please tell me how you got the correct rotation matrix to get the result in the figure?

[thaotrongtran]

You can take a look at the vis_label_in_image.py and vis_utils.py to see the label's path, and how they form the 3D bounding box, and below are the rotation and translation for the frame 000009.

r_velo2cam [[-0.06380332 -0.99109149 -0.04429948] [-0.21028734 0.04399769 -0.79876929] [ 0.97575115 -0.06031493 -0.17158543]] angle 90.28066678210665 89.7638531050683 88.65343846834244 t_velo2cam [[-5.7791444 ] [ 6.03761576] [ 1.0636424 ]] Btw, would you mind sending me an email? I would like to discuss more regarding this. You can find my email on my Github profile. Thanks!

[Devoe-97]

You can take a look at the vis_label_in_image.py and vis_utils.py to see the label's path, and how they form the 3D bounding box, and below are the rotation and translation for the frame 000009.

r_velo2cam [[-0.06380332 -0.99109149 -0.04429948] [-0.21028734 0.04399769 -0.79876929] [ 0.97575115 -0.06031493 -0.17158543]] angle 90.28066678210665 89.7638531050683 88.65343846834244 t_velo2cam [[-5.7791444 ] [ 6.03761576] [ 1.0636424 ]] Btw, would you mind sending me an email? I would like to discuss more regarding this. You can find my email on my Github profile. Thanks!

Thanks. I found something wrong in README that caused my results. When using 'tools/dataset_converter/dair2kitti.py' for roadside data conversion, we should use '--label-type camera' instead of 'lidar' as described in README. My email is '1069498191@qq.com'

[Devoe-97]

You can take a look at the vis_label_in_image.py and vis_utils.py to see the label's path, and how they form the 3D bounding box, and below are the rotation and translation for the frame 000009.

r_velo2cam [[-0.06380332 -0.99109149 -0.04429948] [-0.21028734 0.04399769 -0.79876929] [ 0.97575115 -0.06031493 -0.17158543]] angle 90.28066678210665 89.7638531050683 88.65343846834244 t_velo2cam [[-5.7791444 ] [ 6.03761576] [ 1.0636424 ]] Btw, would you mind sending me an email? I would like to discuss more regarding this. You can find my email on my Github profile. Thanks!

This is still wrong! When using labels under the camera directory, 3d_bboxes can be projected correctly to the corresponding image frames, but these 3d_bboxes are misaligned in the point cloud coordinate system! When using labels under the vituallidar directory, 3d_bboxes are correct in the point cloud coordinate system but can not be projected correctly to the corresponding image frames.

I think this is essentially a misalignment between the camera data frames and the LiDAR data frames.

[thaotrongtran]

You are right, but this is a fundamental issue of time synchronization. There will always be a time difference between the camera frame and the LiDAR frame because the capture frequency is different. You can take a look at the timestamp in the labels they provided.

[hehualin-tut]

是这样的，从激光雷达投影到图像错的离谱，也不知道给出内外参干嘛用的，如果你解决了这个问题，还请劳烦告知我一下，不胜感激。