search

Thinklab-SJTU / Bench2DriveZoo

BEVFormer, UniAD, VAD in CARLA under Closed-Loop Evaluation
44 stars 3 forks source link

Question about the matrix transformation in Bench2DriveZoo/mmcv/datasets/prepare_B2D.py #1

Closed jiangjudging closed 1 week ago

jiangjudging commented 2 weeks ago

In the function preprocess there is one line: 

sensor_infos['LIDAR_TOP']['lidar2ego'] =   np.array(anno['sensors']['LIDAR_TOP']['lidar2ego']) @ lidar_to_righthand_ego

I think lidar_to_righthand_ego means the lidar in nuscenes (x-right,y-forward,z-up) to the right hand ego (x-forward, y-left, z-up) transformation. That is why:

lidar_to_righthand_ego = np.array([[  0, 1, 0, 0],
                                   [  -1, 0, 0, 0],
                                   [  0, 0, 1, 0],
                                   [  0, 0, 0, 1]])

anno['sensors']['LIDAR_TOP']['lidar2ego'] should be the transformation matrix of the lidar in carla to the ego.

So the result of the matrix multiply seems unreasonable. Could you pls check it out or explain for short.

Thanks for your great work.

jiaxiaosong1002 commented 2 weeks ago

@jiangjudging Hi, because in the CARLA official API, the coordinate system is quite different from the nuScenes. All coordinate systems in the model are the same as those codebase in mmdet3d v0.17 (BEVFormer/UniAD/VAD). Thus, it requires some transformation for the raw data.

jiangjudging commented 2 weeks ago

@jiangjudging Hi, because in the CARLA official API, the coordinate system is quite different from the nuScenes. All coordinate systems in the model are the same as those codebase in mmdet3d v0.17 (BEVFormer/UniAD/VAD). Thus, it requires some transformation for the raw data.

Thanks for your quick reply. I got the motivation to make the transform. But just do not think the formular is right. I think the formular should run as follow:

sensor_infos['LIDAR_TOP']['lidar2ego'] = left2right @ np.array(anno['sensors']['LIDAR_TOP']['lidar2ego']) @ np.linalg.inv(left2right) @ lidar_to_righthand_ego

But the value of np.array(anno['sensors']['LIDAR_TOP']['lidar2ego']) is like 

[[  1, 0, 0, x],
[  0, 1, 0, 0],
[  0, 0, 1, z],
[  0, 0, 0, 1]]

which makes left2right @ np.array(anno['sensors']['LIDAR_TOP']['lidar2ego']) @ np.linalg.inv(left2right) == np.array(anno['sensors']['LIDAR_TOP']['lidar2ego']) That is why the code still get the right result. However, once the np.array(anno['sensors']['LIDAR_TOP']['lidar2ego']) changes (like 

[[  1, 0, 0, x],
[  0, 1, 0, y],
[  0, 0, 1, z],
[  0, 0, 0, 1]]

the code might go wrong.

jiaxiaosong1002 commented 2 weeks ago

@jiangjudging Hi, good point! We will fix that. Thanks for your kind advice.