Usually a reprojection error < 1 is a good indicator (but not absolute)

[aaronlsmiles]

    Usually a reprojection error < 1 is a good indicator (but not absolute), please search online you will find plenty of explanations about single camera and stereo camera reprojection error. This is only an issue tracker for the library. However, you need to collect ~100 chessboards to obtain robust results. But this has to be done only once (if you don't move the cameras ;-) ). Cheers.

Originally posted by @decadenza in https://github.com/decadenza/SimpleStereo/issues/5#issuecomment-1255614769

@decadenza I have a few questions relating to your comment (above):

1

Usually a reprojection error < 1 is a good indicator (but not absolute)

I am struggling to get a reprojection error (RPE) <1 .

I am using a 7x6 checkerboard printed on A4 paper and glued to a clipboard, and x2 AKASO EK7000 action cams in 4K 30fps video mode, syncing the left and right video in Premiere Pro, then extracting the still frames*. I have been using ~30 images pairs like your example, but the best RPE I've got so far is 1.29.

I've tested the code with your examples/calib image folders and notice that your images have no distortion, yet mine you can see (below) have barrel distortion. Have your calib image folders been run through the "004a UndistortImages.py" code and should I do the same to my images (then calibrate again)? Is this the practice for improving the RPE, or should I been getting a better initial RPE before doing any undistort actions?

Or, could the fact that I've rescaled the checkerboard be causing issues (as it looks like the checkerboard you use in the calib image folders is larger than mine)?

*I am calibrating this way instead of using your code because I will need to perform this underwater for my research and will not be able to have cameras connected to the computer.

2

However, you need to collect ~100 chessboards to obtain robust results.

Your code uses/suggests 30 images. Should we be collecting 100?

3

Lastly, when you say:

But this has to be done only once (if you don't move the cameras ;-) ).

Do you mean the distance/position between the cameras? I have both cameras fixed to a ruler at 10cm apart. The idea is to move this stereo camera pair around for robot navigation research. To clarify, this will be ok and not require recalibration every time the camera pair move together, correct?

[decadenza]

Although this is not an issue related with the library, I'll try to help for the sake of the research :-).

1 To begin with, reprojection error is not a perfect indicator of the quality of the calibration. This statement is not mine, see this paper on par. IV B.

I used a similar system, but you have to be extra careful that the wet glue does not deform the squares. For my experiments I generally captured ~200 chessboards in several positions. I had good results even with stereo reprojection error of 2.6 in one of my experiments. More squares should be better, but it's easier to have problems (missing a corner during calibration).

N.B Stereo reprojection error consider the pixel displacement that you have going from camera 1 to the chessboard, then to camera 2 and all the way back, so it is normally higher.

Example 004a is just to show how you can correct distortion. You need to start calibration from the original images (not rescaled, not corrected in any way).

The distortion you are seeing depends on the lenses, a good calibration can compensate that, but since the distortion model is... well... a model, it is never capable to correct distortion 100%.

Final though on your system: using the video feed may cause some blur (video is generally compressed, try to deactivate interlacing). You may find a way to take photos or integrating the video feed directly in the library.

TL;DR Don't care too much about RPE, it's an indicator (that depends also on pixel sensor size). The best thing is to try to scan something with known measures.

2 Yes, in my experience more image of chessboards collected in different position lead to a better calibration (even if the reprojection error may increase).

3 Sure. If there is no relative movement between the cameras, the calibration parameters remain the same, otherwise extrisincs do change.

A question from me now: doesn't water and refraction affect the image? Did you test the system underwater.

[aaronlsmiles]

Thanks @decadenza that's really useful knowledge! 💯 :-)

Just out of curiosity, what cameras did you use for stereo calibration and what size paper is the chessboard printed on (A3?)? Does the orientation of the checkerboard matter (eg. 7x6 vs 6x7, or 7x6 portrait vs landscape?)?

Water/refraction will affect the image, but this is part of what I will be investigating and trying to optimise. I will be calibrating with new images underwater with a laminated checkerboard. Right now I am just trying to get the system to work above water. Will keep you posted of results if you're interested!

[decadenza]

The size of the chessboard matters in relation to the camera resolution, field of view and distance. As a rule of thumb, if your camera is low resolution, makes sense to have few squares, and vice versa. If your camera has a large field of view, makes sense to use a bigger chessboard. But don't trust my opinion too much.

I used 9 x 6 printed over A4 and glued on an aluminium board. I used different cameras, often ELP USB ones.

Really nice project. I am glad this library is useful, probably you can adapt it and maybe make your code work on a Raspberry Pi too!

Issue closed. Good luck.