Stereo Image Rectification

[cxy1997]

Hi, I found that the stereo images in Argoverse are slightly misaligned.

I tried to rectify them with

left_img, right_img = cv2.imread(left_src), cv2.imread(right_src)
R = np.matmul(np.linalg.inv(calibL.R), calibR.R)
T = np.matmul(np.linalg.inv(calibL.R), (calibR.T - calibL.T))
distCoeff = np.zeros(4)

R1, R2, P1, P2, Q, roi1, roi2 = cv2.stereoRectify(
    cameraMatrix1=calibL.K[:3, :3],
    distCoeffs1=distCoeff,
    cameraMatrix2=calibR.K[:3, :3],
    distCoeffs2=distCoeff,
    imageSize=(STEREO_IMG_WIDTH, STEREO_IMG_HEIGHT),
    R=R,
    T=T,
    flags=cv2.CALIB_ZERO_DISPARITY,
    alpha=0
)

map1x, map1y = cv2.initUndistortRectifyMap(
    cameraMatrix=calibL.K[:3, :3],
    distCoeffs=distCoeff,
    R=R1,
    newCameraMatrix=P1,
    size=(STEREO_IMG_WIDTH, STEREO_IMG_HEIGHT),
    m1type=cv2.CV_32FC1)

map2x, map2y = cv2.initUndistortRectifyMap(
    cameraMatrix=calibR.K[:3, :3],
    distCoeffs=distCoeff,
    R=R2,
    newCameraMatrix=P2,
    size=(STEREO_IMG_WIDTH, STEREO_IMG_HEIGHT),
    m1type=cv2.CV_32FC1)

left_img_rect = cv2.remap(left_img, map1x, map1y, cv2.INTER_LINEAR)
right_img_rect = cv2.remap(right_img, map2x, map2y, cv2.INTER_LINEAR)
cv2.imwrite(left_dst, left_img_rect)
cv2.imwrite(right_dst, right_img_rect)

The rectified images are still misaligned.

Have you tried to rectify the images? I'm not sure if my code is wrong or if the calibration parameters are incorrect. Thanks a lot!

[James-Hays]

Are you experimenting with Argoverse 1.0 or 1.1 (see https://www.argoverse.org/data.html#download-link )? In version 1.1 we reduced the epipolar alignment error by half.

[cxy1997]

Hi @James-Hays, we are using Argoverse 1.1.

[johnwlambert]

Hi @cxy1997, I'm looking at the documentation for initUndistortRectifyMap and it looks like you are computing a joint undistortion and rectification transform. That would be helpful for distorted imagery, but Argoverse imagery has already been undistorted, so that pixel remapping transform won't be right.

[cxy1997]

Hi @johnwlambert, I don't think that's the cause of the problem. I have set all distortion parameters to zero so they won't influence the transform.

[dikpalargo]

@cxy1997 Can you please check if the T you computed in line 3 in your code snippet is correct? I think it should be T_r - R * T_l

[johnwlambert]

@cxy1997 Got it, sounds reasonable then if distortion coefficients are set to zero.

@dikpalargo I'm also concerned about the transformation @cxy1997 calculated between camera frames. We provide a transformation from the egovehicle frame (center of back axle) to the camera coordinate frame cam_SE3_egovehicle that will take cam_pt = cam_SE3_egovehicle * p_egovehicle I would expect to do something like camL_SE3_egovehicle.dot(inv(camR_SE3_egovehicle )) to get Right->Left transformation I would expect something like:

[cxy1997]

Hi @johnwlambert, great suggestion!

I recalculated R and T with

extrinsic = np.dot(calibR.extrinsic, np.linalg.inv(calibL.extrinsic))
R = extrinsic[:3, :3]
T = extrinsic[:3, 3]

since cv2.stereoRectify() needs Left -> Right transformation matrix

The stereo images are aligned now

Thanks so much!

[johnwlambert]

No problem @cxy1997, let me know if you run into any other difficulties with the data or API code.

[lizolson]

Hi @johnwlambert, have all of the datasets for 1.1 been updated? The sequences in the first folder (train1) are different between 1.0 and 1.1. For the rest of the training folders, the sequences are the same, and the stereo calibrations appear to be as well. Thanks!

[alliecc]

Hi @lizolson1, many of the logs in Argoverse1.1 are updated. Some have updated labels and some have updated stereo calibration, and the logs in train and val are randomly reselected. We recommend to use Argoverse 1.1 and not 1.0. Sorry for the inconvenience!

[lizolson]

Thank you @alliecc!!

[swdev1202]

Thank you guys for sharing a great example! I am fairly new to computer vision and was not understanding the transformation between reference frames, etc. This video gives an awesome explanation of what @johnwlambert was showing on the paper! :)