Getting the pose of the camera for each frame

[fvgt]

Hi,

I would like to know how I can obtain the pose of the camera for every frame. As far as I understand, the poses specified in the 'ego_pose' file are the pose of the vehicle. My use case is that I have points in the camera coordinate system, and I want to adjust the pose of the camera. So what I would need is the calibration from the camera w.r.t. to the ego frame (i.e., transform the camera coordinates to the ego coordinates), apply the pose to the points (the pose given in the 'ego_pose' file), and then transform the coordinates back to the camera, (using the inverse of the camera to ego transformation), is that correct?

[whyekit-motional]

@fvgt to get the pose of the camera in every keyframe, you can do something like this:

from nuscenes.nuscenes import NuScenes

nusc = NuScenes(version='v1.0-mini', dataroot='/data/sets/nuscenes', verbose=False)

sample = nusc.sample[0]
sample_data = nusc.get('sample_data', sample['data']['CAM_FRONT'])
calibrated_sensor = nusc.get('calibrated_sensor', sample_data['calibrated_sensor_token'])
print(calibrated_sensor)

The above would give the following, which should contain the info you require:

{
    'token': '1d31c729b073425e8e0202c5c6e66ee1', 
    'sensor_token': '725903f5b62f56118f4094b46a4470d8', 
    'translation': [1.70079118954, 0.0159456324149, 1.51095763913], 
    'rotation': [0.4998015430569128, -0.5030316162024876, 0.4997798114386805, -0.49737083824542755], 
    'camera_intrinsic': [
        [1266.417203046554, 0.0, 816.2670197447984], 
        [0.0, 1266.417203046554, 491.50706579294757], 
        [0.0, 0.0, 1.0]
    ]
}

[fvgt]

Thank you very much! I am just wondering why you are using only the calibrated sensor token, I was assuming that I also need the pose of the vehicle at the given time frame. In other words, I thought I would need the 'ego_poses'

[whyekit-motional]

@fvgt yes, in addition to the pose of the camera wrt to the ego, you will need the ego pose itself - I will leave it to you as an exercise to figure out how to retrieve it (as a clue: there is ego_pose_token within sample_data - see the schema)

[fvgt]

@whyekit-motional I give it a try :)

sample = nusc.sample[0]
cam2ego = load_cam_to_ego(nuscenes, sample) # this is basically your code above
ego_pose_info = nuscenes.get("ego_pose",  nuscenes.get("sample_data", sample["data"]['CAM_FRONT])["ego_pose_token"])  
ego_pose =  make_transform_matrix(ego_pose_info["rotation"], ego_pose_info["translation"])
cam_pose = (np.linalg.inv(cam2ego) @ ego_pose @ cam2ego) # transform camera to ego, to apply the pose and then back to camera

Some function are implemented behind the scenes, like load_cam_to_ego and make_transform_matrix, but it should be clear from the context what they do. What do you think?

[whyekit-motional]

@fvgt you could use this as a reference: https://github.com/nutonomy/nuscenes-devkit/blob/9b165b1018a64623b65c17b64f3c9dd746040f36/python-sdk/nuscenes/nuscenes.py#L298-L300

It moves a box from the ego frame to the sensor frame, but it should be trivial to modify it to your use case of moving a point in the sensor (camera, in your case) frame to the ego frame