Fair results from individual calibration, but bad results from --pairs

[zumpchke]

I'm unable to get good stereo results using mrcal from ubuntu jammy.

• Camera 1 is an IR CAM with default output 512 x 424 made to fit 1920 x 1080 (aspect ratio NOT adhered to. so there is stretching). It is semi-wide angle (70.6 x 60 degrees)
• Camera 2 approximates a pinhole setup with resolution 1920 x 1080

I'm scaling camera 1 because it doesn't seem possible for 2 cameras to have different image sizes using --pairs

When I calibrate each camera individually, I get fair results and --show-geometry looks okay. But it goes out of whack when using --pairs to get the extrinsics.

I have ~11 board views. I could get more, but I think something is systematically wrong.

My ultimate goal is to be able to take an image in camera 1 view and transform it to camera 2 taking into account all distortions and geometry

Camera 1 cal

# mrcal-calibrate-cameras --lensmodel LENSMODEL_OPENCV8 --object-spacing 0.05 --object-width-n 10 --object-height-n 7  --imagersize 1920 1080 --corners-cache test/corners.vnl --outdir test/ --focal 1900  cam1_*.png
## initial solve: geometry only
## RMS error: 4.88

## initial solve: geometry and LENSMODEL_STEREOGRAPHIC core only
=================== optimizing everything except board warp from seeded intrinsics
## final, full optimization
## RMS error: 1.38
RMS reprojection error: 1.4 pixels
Worst residual (by measurement): 6.7 pixels
Noutliers: 0 out of 770 total points: 0.0% of the data
calobject_warp = [0.00027107 0.00082771]

Wrote test//camera-0.cameramodel

Camera 2 cal

# mrcal-calibrate-cameras --lensmodel LENSMODEL_OPENCV8 --object-spacing 0.05 --object-width-n 10 --object-height-n 7  --imagersize 1920 1080 --corners-cache test/corners.vnl --outdir test/ --focal 1900  cam2_*.png
## initial solve: geometry only
## RMS error: 4.32

## initial solve: geometry and LENSMODEL_STEREOGRAPHIC core only
=================== optimizing everything except board warp from seeded intrinsics
mrcal.c(5564): Threw out some outliers. New count = 6/1540 (0.4%). Going again
## final, full optimization
## RMS error: 0.59
RMS reprojection error: 0.6 pixels
Worst residual (by measurement): 2.9 pixels
Noutliers: 6 out of 770 total points: 0.8% of the data
calobject_warp = [-0.00087069 -0.00044855]

Wrote test//camera-0.cameramodel

Stereo cal

# mrcal-calibrate-cameras --lensmodel LENSMODEL_OPENCV8 --object-spacing 0.05 --object-width-n 10 --object-height-n 7  --imagersize 1920 1080 --corners-cache test/corners.vnl --outdir test/ --pairs --focal 1900  cam1_*.png cam2_*.png
## initial solve: geometry only
## RMS error: 60.47

## initial solve: geometry and LENSMODEL_STEREOGRAPHIC core only
=================== optimizing everything except board warp from seeded intrinsics
## final, full optimization
## RMS error: 47.50
RMS reprojection error: 47.5 pixels
Worst residual (by measurement): 211.0 pixels
Noutliers: 0 out of 1540 total points: 0.0% of the data
calobject_warp = [ 3.3806449e-09 -4.7381047e-09]

Wrote test//camera0-0.cameramodel
Wrote test//camera0-1.cameramodel

Here is my corners file, I'm unable to share the original images

corners.vnl.zip

[zumpchke]

Also, is there a way to get extrinsics with two .cameramodels (without using --pairs) ?

[dkogan]

Hello. First of all, --pairs only affects the output filenames; that's it. Nothing you're doing needs it or will be broken by it. Also, mrcal doesn't care about the imager dimensions matching up. So you should use the actual image size. If that doesn't work, tell me.

Your joint solve doesn't work because there's something wrong with your input data. I can't tell without spending more time on it, but I bet the issue isn't subtle. As you see, the data does fit if you look at it monocularly. You can use those intrinsics and ask it to estimate extrinsics only:

mrcal-calibrate-cameras --lensmodel LENSMODEL_OPENCV8 --object-spacing 0.05 --object-width-n 10 --object-height-n 7  --imagersize 1920 1080 --corners-cache corners.vnl --seed mono/camera-0.cameramodel,mono/camera-1.cameramodel --skip-intrinsics-solve 'cam'{1,2}'_*.png'

It doesn't fit. And the best-fit geometry looks suspicious (run mrcal-show-geometry --show-calobjects camera-[01].cameramodel; the best-fit solve has the cameras pointed at each other, looking at the chessboards from opposite sides). What is the actual geometry of the two cameras?

Your data looks weird and wrong. Do this:

(echo '# filename x y'; < corners.vnl) | vnl-filter 'filename == "cam1_1.png"' -p x,y | feedgnuplot --domain --lines --points --square

This visualizes the corner detections. They're very irregular and jagged for the IR camera. Your corners need to be observable better; this is a smaller problem you'll need to fix after you fix the big problem. In any case, if you visualize the corners for cameras 1 and 2 using the above command, you'll see that they truly are reporting the corners in the opposite order: cam1 has the first point at the bottom-left, then it goes right and up; cam2 has the first point at the top-left, then it goes right and down. That's your problem and that's why mrcal has the cameras looking in from opposite directions. You should look at your corner detections, and make sure that they are right and consistent.

[zumpchke]

Thank you! the ordering of cam1 was definitely an issue, and I'm trying to improve the quality of the cam1 corners too.

One more question - if trying to solve for extrinsics between cam1 and cam2, does each observation for the respective camera have to be of the same view of the calibration object?

[dkogan]

Not EACH observation. But you must have SOME instances in time where multiple cameras can simultaneously observe the object.