[Feature request] Rectification of stereo image pairs.

[copaah]

🚀 Feature

To complete the stereo pipeline in Kornia, we need functionality for rectifying images captured by a stereo camera setup.

This will be somewhat similiar to the OpenCV function cv2.initUndistortRectifyMap, but I don't think we would initialize the rectification maps first. Instead we would just have the signature be:

def rectify(image, camera_matrix, R))
     return ...

def rectify_stereo_pair(left_image_batch, right_image_batch, left_camera_matrix, right_camera_matrix, R_left, R_right):
     left_image_batch_rectified = rectify(left_image_batch, left_camera_matrix, R_left)
     right_image_batch_rectified  = rectify(right_image_batch, right_camera_matrix, R_right)
     return left_image_batch_rectified, right_image_batch_rectified

Optionally we would also return the rectified camera matrices.

Note that OpenCV will undistortion and rectification in one step. I am unsure if this should be part of this first feature, but might be the way to go.

The kornia.geometry.undistort_points might be helpful here.

Motivation

To complete the stereo pipeline in Kornia, we need rectification functionality. That will allow the full stereo pipeline: Bayer Image -> Color Image -> Undistort -> Rectify -> Disparity map -> Point Cloud to be supported by Kornia.

Pitch

See above. I need to find good material for a rectification algorithm.

Here's a starting point:

https://www.cs.cmu.edu/~16385/s17/Slides/13.1_Stereo_Rectification.pdf

[edgarriba]

Wondering how we could use here the differentiability properties form pytorch and for example how could be implemented seamless to the Image Registration algorithm https://kornia.readthedocs.io/en/latest/applications/image_registration.html

[copaah]

@edgarriba

I think this is one of the more important tasks for the stereo module, so I can try to take it on. Will need some help outlining the algorithm though and also some guidance on how to integrate with the Image Registration API.

[edgarriba]

my guess here is to try implement standalone and later we see how can be integrated with the registration api API. As initial approach I would try to implement in terms of plane sweep algorithm by building a cost volume. In fact, that was the very first initial use case of kornia and the HomographyWarper. /cc @ducha-aiki any thoughts here ?

[ducha-aiki]

It is very needed indeed! I agree that the best way is to create such functions alone and integrate later.

[copaah]

From talking with @edgarriba there are two sets of functionality we want to support akin to those in OpenCV:

• stereoRectify
• stereoRectifyUncalibrated

My initial idea was to go ahead with implementing something similar to stereoRectify. I think @edgarriba was talking about tackling the other use case.

[edgarriba]

@copaah my idea was to implement the local feature matching pipeline in the library and add the solver for the homographies that rectifies the epipolar lines. @ducha-aiki proposed to implement this paper: https://hal-enpc.archives-ouvertes.fr/hal-00654415/document in case we don't find a direct implementation. Was checking into OpenCV code but they don't provide references of which method they implement AND no comments in the code which makes quite difficult to decode what's happening in there.

[copaah]

I'll give the algorithm in the attached document a shot. It's for rectifying images with known stereo calibration.

vdocuments.mx_a-compact-algorithm-for-rectification-of-stereo-pairs.pdf s

[edgarriba]

Looks like a good one

[copaah]

Added first go a reimplementing the algorithm from the paper. I get approximately same results, but need data from a real stereo rig to verify.

[alonks1234]

Hi, has the feature stalled? Thinking about putting together a deep differentiable stereo reconstruction pipeline and this would greatly simplify it.

[edgarriba]

@alonks1234 what method are you planning ?

[alonks1234]

The pipeline would be kornia to undistorted images, unknown to rectify, then raft-stereo to match the rectified images ind produce a disparity and kornia again for 3d projection into a point cloud. Because it is differentiable end-to-end it could open up interesting possibilities such as optimizing additional losses, refining the intrinsics/extrinsics, etc.

[edgarriba]

Is Raft fast now ? Last time I checked was kinda slow for real time applications or edge devices. Could you point to the implementation you plan to port ? Just to check license is good and if weights are okay too

[alonks1234]

This may well not run on an edge device.

https://github.com/princeton-vl/RAFT-Stereo

And also may go a different direction for something lighter weight. But either way I'll need to rectify stereo images, and would like to understand the status of this feature.

[copaah]

Haven't worked more on this. Welcoming any contributions.