The meaning of calibration distortion in nuImages

[ZwwWayne]

Hi developers,

Thanks for the great work on nuImages. I have a question about the calibration distortion parameters.

In your documentation in the official page, you says that

We use the 5 parameter camera convention of the CalTech camera calibration toolbox, that is also used in OpenCV. Only for fish-eye lenses in CAM_BACK do we use the 6th parameter.

I am a little bit confused about their physical meanings, i.e., the exact mappings from the 5/6 parameters to the coefficients k1, k2, k3, k4, k5, k6, p1, p2 in the OpenCV documentation. As the correct order in OpenCV is (k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6]]), should I simply treat them as from k_1 to k_5/k_6 or treat them as the first 5 or 6 parameters in the OpenCV order?

For now, I tried to undistort the image as below:

    calibrated_sensor = nuim.get(
        'calibrated_sensor',
        sample_data['calibrated_sensor_token'])
    cam_intrinsic = calibrated_sensor['camera_intrinsic']
    cam_distortion = calibrated_sensor['camera_distortion']
    cam_intrinsic = np.array(cam_intrinsic, dtype=np.float32)
    cam_distortion = np.array(cam_distortion, dtype=np.float32)
    print(f'cam_intrinsic: {cam_intrinsic}')
    print(f'cam_distortion: {cam_distortion}')
    newcameramtx, roi = cv2.getOptimalNewCameraMatrix(
        cam_intrinsic,
        cam_distortion,
        (width, height), 1, (width, height))

The above code works in some situations. However, when it prints cam_distortion: [-3.01740e-01 1.38439e-01 3.31000e-04 -8.00000e-06 -4.73840e-02 7.45000e-03], the OpenCV throws an error as below:

cv2.error: OpenCV(4.2.0) /io/opencv/modules/calib3d/src/undistort.dispatch.cpp:297: 
error: (-215:Assertion failed) CV_IS_MAT(_distCoeffs) && 
(_distCoeffs->rows == 1 || _distCoeffs->cols == 1) && 
(_distCoeffs->rows*_distCoeffs->cols == 4 || _distCoeffs->rows*_distCoeffs->cols == 5 || _distCoeffs->rows*_distCoeffs->cols == 8 || _distCoeffs->rows*_distCoeffs->cols == 12 || _distCoeffs->rows*_distCoeffs->cols == 14) 
in function 'cvUndistortPointsInternal'

So I am confused about the correct implementation, could you help me to undistort the images correctly?

[holger-motional]

Hi. Please note this sentence on the homepage: "Contrary to nuScenes, all camera images come distorted and unrectified". So most users may not actually want to undistort the images. If you do, please take a look at this old code snippet that we never released: https://github.com/nutonomy/nuscenes-devkit/blob/00bfe1e9d93eca172ab651559b34e6cc3d953038/python-sdk/nuimages/utils/lidar.py#L63 It shows how to project a pointcloud into the image (we did not publish the pointclouds as they had some issues). The parameters are: k1 = camera_distortion[0] k2 = camera_distortion[1] p1 = camera_distortion[2] p2 = camera_distortion[3] k3 = camera_distortion[4] k4 = camera_distortion[5]

[ZwwWayne]

Great! Thanks for your kind reply.

I am debugging the model's evaluation performance when they are trained on nuImages and evaluated on nuScenes, and I am considering whether the calibration is the issue yesterday so I want to have a try. Now I think it is probably not the calibration issue but the information you provide is very useful.