Question about nuscenes camera2lidar extrinsic

[lucianzhong]

For one scenario in nuscenes dataset, take 61 as example, the sensor2lidar_translation between CAM_FRONT and lidar_top differs from one timestamp to another timestamp. such as: [-0.01613824 0.43552529 -0.32067179], [-0.00871988 0.5708719 -0.31998898] 0.5708719 - 0.43552529 = 0.13, the y direction of translation is the vehicle's forward direction. From my opinion,the extrinsic should ideally remain constant or minor variation. Could someone give explanation?

Best regards

[whyekit-motional]

@lucianzhong pls provide a working code snippet to reproduce this issue

[lucianzhong]

def obtain_sensor2top(nusc,
                      sensor_token,
                      l2e_t,
                      l2e_r_mat,
                      e2g_t,
                      e2g_r_mat,
                      sensor_type='lidar'):
    """Obtain the info with RT matric from general sensor to Top LiDAR.
    Args:
        nusc (class): Dataset class in the nuScenes dataset.
        sensor_token (str): Sample data token corresponding to the
            specific sensor type.
        l2e_t (np.ndarray): Translation from lidar to ego in shape (1, 3).
        l2e_r_mat (np.ndarray): Rotation matrix from lidar to ego
            in shape (3, 3).
        e2g_t (np.ndarray): Translation from ego to global in shape (1, 3).
        e2g_r_mat (np.ndarray): Rotation matrix from ego to global
            in shape (3, 3).
        sensor_type (str): Sensor to calibrate. Default: 'lidar'.
    Returns:
        sweep (dict): Sweep information after transformation.
    """
    sd_rec = nusc.get('sample_data', sensor_token)

    print("calibrated_sensor_token  ",sd_rec['calibrated_sensor_token'])

    cs_record = nusc.get('calibrated_sensor',
                         sd_rec['calibrated_sensor_token'])

    pose_record = nusc.get('ego_pose', sd_rec['ego_pose_token'])

    data_path = str(nusc.get_sample_data_path(sd_rec['token']))

    if os.getcwd() in data_path:  # path from lyftdataset is absolute path
        data_path = data_path.split(f'{os.getcwd()}/')[-1]  # relative path
    sweep = {
        'data_path': data_path,
        'type': sensor_type,
        'sample_data_token': sd_rec['token'],
        'sensor2ego_translation': cs_record['translation'],
        'sensor2ego_rotation': cs_record['rotation'],
        'ego2global_translation': pose_record['translation'],
        'ego2global_rotation': pose_record['rotation'],
        'timestamp': sd_rec['timestamp']
    }
    l2e_r_s = sweep['sensor2ego_rotation']
    l2e_t_s = sweep['sensor2ego_translation']
    e2g_r_s = sweep['ego2global_rotation']
    e2g_t_s = sweep['ego2global_translation']

    print("l2e_r_s  ",l2e_r_s)
    print("l2e_t_s   ",l2e_t_s)
    print("e2g_r_s ",e2g_r_s)
    print("e2g_t_s  ",e2g_t_s)

    # obtain the RT from sensor to Top LiDAR
    # sweep->ego->global->ego'->lidar

    l2e_r_s_mat = Quaternion(l2e_r_s).rotation_matrix
    e2g_r_s_mat = Quaternion(e2g_r_s).rotation_matrix

    R = (l2e_r_s_mat.T @ e2g_r_s_mat.T) @ (
        np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(l2e_r_mat).T)

    T = (l2e_t_s @ e2g_r_s_mat.T + e2g_t_s) @ (
        np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(l2e_r_mat).T)

    T -= e2g_t @ (np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(l2e_r_mat).T
                  ) + l2e_t @ np.linalg.inv(l2e_r_mat).T
    sweep['sensor2lidar_rotation'] = R.T  # points @ R.T + T

    sweep['sensor2lidar_translation'] = T

    print("sensor_type  ",sensor_type)
    print("sensor2lidar_rotation  ",R.T)
    print("sensor2lidar_translation  ",T)
    return sweep

[lucianzhong]

@whyekit-motional please have a look

[whyekit-motional]

@lucianzhong try using this part of the devkit as a reference to write your transformation method instead: https://github.com/nutonomy/nuscenes-devkit/blob/a5c089133baa001d3ab3c5583a103957e4ae8375/python-sdk/nuscenes/nuscenes.py#L881-L890