Added support for fisheye (and mixed fisheye/pinhole) stereo rigs

[KevinCain]

I added fisheye calibration to my fork of SimpleStereo (master), following notes here and also here.

In 'chessboardStereo' there are two new calling parameters which allow you to call the method with one or two fisheye cameras:

fisheye1, fisheye2 : bool
    Set to true if one or both of the stereo cameras have fisheye lenses
    For fisheye cameras, the number of distortion coefficients is 4 instead of 5.

For a stereo rig where one camera has a fisheye lens and the other has a regular lens, we initialize and pass intrinsic parameters (cameraMatrix, distCoeffs) obtained via ‘cv2 .fisheye.calibrate’ for the fisheye lens, while for the regular camera we continue to use ‘cv2.calibrateCamera’, which seems designed to accommodate different camera models and distortions as long as the appropriate intrinsic parameters (cameraMatrix, distCoeffs) are provided for each camera.

rig = ss.calibration.chessboardStereo(images, chessboardSize=(7,6), squareSize=52.0, fisheye1=1, fisheye2=0) v. rig = ss.calibration.chessboardStereo(images, chessboardSize=(7,6), squareSize=52.0, fisheye1=0, fisheye2=0)

We carry the same handling to the ‘undistortImages’ class in order to handle both fisheye and pinhole camera models. When undistorting images from a fisheye lens, we call ‘cv2.fisheye.initUndistortRectifyMap’ function to compute the undistortion and rectification transformation maps, then apply these maps to the input image using ‘cv2.remap’.

The reprojection results are 2.6 pixels from a small group of (15) chessboard pairs, which you can download here with 'BuildStereoRig.py' to run the calibration, and 'display_images.py' to display results, and the SimpleStereo rigs run with and without fisheye handling.

Since our images don’t adequately cover the field of view, the distortion correction for the area outside the chessboard degrades rapidly. The framing of the chessboard shown is necessary for this reason: one camera tilts away from the other, reducing the overlap between the two cameras as described here.

The results here are identical with or without the fisheye handling, which may be because the normal cv methods work with these images (in the small local region of the frame where we have the chessboard), or there is an implementation problem I'm not seeing.

Note that if you have mixed fisheye and pinhole camera images, choosing to set both as fisheye causes arcane errors in 'cv2.fisheye.stereoCalibrate'.

Here is a sample stereo pair with input on the left and output on the right:

[decadenza]

Hi @KevinCain,

Thank you for sharing this.

The epipolar lines do not seem to match, though. The first of the left image should be passing through the same area of the chessboard. In your code to display the images I see you are resizing the images, that could be the cause.

import sys
import os

import cv2
import numpy as np

import simplestereo as ss
"""
Display images
"""

# Paths
curPath = os.path.dirname(os.path.realpath(__file__))
imgPathL = os.path.join(curPath, 'revok', 'chessboard_d', 'leftb (3).pgm')
imgPathR = os.path.join(curPath, 'revok', 'chessboard_d', 'rgbb (3).jpg')

# StereoRig file
loadFile = os.path.join(curPath,"revok","rig.json")

# Load stereo rig from file
rig = ss.StereoRig.fromFile(loadFile)

print("Image path L:", imgPathL)
print("Image path R:", imgPathR)

# Read right and left image (please ensure the order!!!)
img1 = cv2.imread(os.path.join(imgPathL))
resized_img1 = cv2.resize(img1, (512, 512)) # <--- **Resizing image affects intrinsic parameters!!!**
img2 = cv2.imread(os.path.join(imgPathR))
resized_img2 = cv2.resize(img2, (512, 512)) # <--- **Resizing image affects intrinsic parameters!!!**

# Show images
cv2.imshow('L', resized_img1)
cv2.imshow('R', resized_img2)

# Undistort two images
img1, img2 = rig.undistortImages(img1, img2)

# Your 3x3 fundamental matrix
F = rig.getFundamentalMatrix()

# Number of evenly spaced epipolar lines
N = 10

# Height and width of img1
height1, width1 = img1.shape[0], img1.shape[1]

# Compute the y-coordinates at which the lines will be drawn
y_coords = np.linspace(0, height1, N, endpoint=False)

# Choose the x-coordinate as the midpoint of the width of img1 for all points
x_coord = width1 // 2

# Create the list of points
x1_points = [(x_coord, int(y)) for y in y_coords]

# Call the drawCorrespondingEpipolarLines function
ss.utils.drawCorrespondingEpipolarLines(img1, img2, F, x1=x1_points, x2=[], color=(0, 0, 255), thickness=3)

# **You may resize here for displaying purposes only, *after* drawing the lines.**
resized_img1_u = cv2.resize(img1, (512, 512)) 
resized_img2_u = cv2.resize(img2, (512, 512))

# Show images
cv2.imshow('img1 Undistorted', resized_img1_u)
cv2.imshow('img2 Undistorted', resized_img2_u)
cv2.waitKey(0)
cv2.destroyAllWindows()

print("Done!")

Anyway thank you for your contribution. I'll review it and eventually merge it with simple stereo in the following days.

[KevinCain]

Thanks, @decadenza,

I believe the code I checked in relating to fisheye camera handling is all right, but I see methods that need to be updated to handle fisheyes, for example computing F fails in the above script at the line:F = rig.getFundamentalMatrix() in the above script.

As you know, there are a few issues:

• In the pinhole camera model, it's straightforward to compute the fundamental matrix from the intrinsic and extrinsic parameters of the stereo rig, but of course fisheye distortions violate the assumptions of the pinhole model.
• We run into trouble reusing the existing ss methods as fisheye lenses (usually?) use a 4-parameter distortion model, while pinhole models here use a 5 or 8-parameter model.
• In ss utils np.float is used: I think that is a deprecated alias for the builtin float; I had to replace with np.float64 and rebuild the library to avoid errors.

Here's my edit for the above code, which undistorts points via 'cv2.fisheye.undistortPoints' for the fisheye image before computing F, but the results are still not useful.

import sys
import os

import cv2
import numpy as np

import simplestereo as ss
"""
Display images
"""

# Paths
curPath = os.path.dirname(os.path.realpath(__file__))
imgPathL = os.path.join(curPath, 'revok', 'chessboard_d', 'leftb (3).pgm')
imgPathR = os.path.join(curPath, 'revok', 'chessboard_d', 'rgbb (3).jpg')

# StereoRig file
loadFile = os.path.join(curPath,"revok","rig.json")

# Load stereo rig from file
rig = ss.StereoRig.fromFile(loadFile)

# Read right and left image (please ensure the order!!!)
img1 = cv2.imread(os.path.join(imgPathL))
img2 = cv2.imread(os.path.join(imgPathR))

# Fisheye undistortion here
pts1 = np.array([[(512 // 2, 512 // 2)]], dtype=np.float32)  # Replace with actual points if available
pts2 = np.array([[(512 // 2, 512 // 2)]], dtype=np.float32)  # Replace with actual points if available

# Fisheye undistortion for the first image
if rig.intrinsic1.shape == (3, 3) and rig.intrinsic1.dtype in [np.float32, np.float64]:
    if len(rig.distCoeffs1) == 4:
        undistorted_pts1 = cv2.fisheye.undistortPoints(pts1, rig.intrinsic1, rig.distCoeffs1)
    else:
        print("Error: Length of distortion coefficients for the first camera must be 4.")
else:
    print("Error: Intrinsic matrix for the first camera should be of size 3x3 and type float32 or float64.")

# Pinhole undistortion for the second image
if rig.intrinsic2.shape == (3, 3) and rig.intrinsic2.dtype in [np.float32, np.float64]:
    if len(rig.distCoeffs2) in [5, 8]:  # Checking for either 5 or 8 coefficients
        undistorted_pts2 = cv2.undistortPoints(pts2, rig.intrinsic2, rig.distCoeffs2)
    else:
        print("Error: Length of distortion coefficients for the second camera must be 5 or 8.")
else:
    print("Error: Intrinsic matrix for the second camera should be of size 3x3 and type float32 or float64.")

# Undistort two images
img1, img2 = rig.undistortImages(img1, img2)

# Your 3x3 fundamental matrix
F = rig.getFundamentalMatrix()

# Number of evenly spaced epipolar lines
N = 10

# Height and width of img1
height1, width1 = img1.shape[0], img1.shape[1]

# Compute the y-coordinates at which the lines will be drawn
y_coords = np.linspace(0, height1, N, endpoint=False)

# Choose the x-coordinate as the midpoint of the width of img1 for all points
x_coord = width1 // 2

# Create the list of points
x1_points = [(x_coord, int(y)) for y in y_coords]

# Call the drawCorrespondingEpipolarLines function
ss.utils.drawCorrespondingEpipolarLines(img1, img2, F, x1=x1_points, x2=[], color=(0, 0, 255), thickness=3)

# **You may resize here for displaying purposes only, *after* drawing the lines.**
resized_img1_u = cv2.resize(img1, (512, 512)) 
resized_img2_u = cv2.resize(img2, (512, 512))

# Show images
cv2.imshow('img1 Undistorted', resized_img1_u)
cv2.imshow('img2 Undistorted', resized_img2_u)
cv2.waitKey(0)
cv2.destroyAllWindows()

print("Done!")

[KevinCain]

Above I showed a calibration attempt between one camera with ~120^ FOV and another camera with ~90^ FOV. If I use two identical ~120^ FOV cameras the reprojection error drops to 0.127 and the epipolar lines look sane:

Here's the rectified stereo pair:

From this I suppose it's clear that OpenCV not only doesn't require fisheye methods for ~120^ FOV, but in fact using them above causes problems, at least how I am implementing the OpenCV fisheye methods.

[KevinCain]

I added standalone python files 'pinhole.py' and 'fisheye.py' to my ss fork.

These perform OpenCV chessboard detection/calibration/reprojection without using the SimpleStereo library, as a sanity check.

The SimpleStereo reprojection error for the same data set is: 0.127, as above. Reprojection error for 'pinhole.py' here is:

L: 0.001572786135453764 pixels
R:0.0019744146026290967 pixels

Both are quite good. I haven't tried to account for the differences.

[124bit]

Adding fisheye functionality to the lib will be very useful

[decadenza]

It can be done by storing the type of cameras in the rig and conditionally performing all calibration and rectification. Or better, creating a separate type of rig. If you'd like to give it a go, share/pull request your results please!