Testing results are not good as expected according to the paper

[xoox]

We've conducted tests of this calibrating method with test data of calibrel_testdata. The results are not as good as expected according to the paper. The new method is sometimes worse than OpenCV's method and sometimes better than the latter. But in general there is no significant improvements compared to OpenCV's standard method. The detailed test results are showed following.

Data set 1

OpenCV's method

<camera_matrix type_id="opencv-matrix">
  <rows>3</rows>
  <cols>3</cols>
  <dt>d</dt>
  <data>
    3.0670601055435177e+03 0. 5.4168384780123540e+02 0.
    3.0737503993310925e+03 5.3691339052892795e+02 0. 0. 1.</data></camera_matrix>
<distortion_coefficients type_id="opencv-matrix">
  <rows>5</rows>
  <cols>1</cols>
  <dt>d</dt>
  <data>
    -1.6114330409319458e-01 0. -2.2729685301187963e-03
    -3.6581288200174624e-03 0.</data></distortion_coefficients>
<avg_reprojection_error>3.1406583963136664e-01</avg_reprojection_error>

DLR11 method

<camera_matrix type_id="opencv-matrix">
  <rows>3</rows>
  <cols>3</cols>
  <dt>d</dt>
  <data>
    3.0209501913836739e+03 0. 6.3993993777133790e+02 0.
    3.0271840392465974e+03 5.1194779030932455e+02 0. 0. 1.</data></camera_matrix>
<distortion_coefficients type_id="opencv-matrix">
  <rows>5</rows>
  <cols>1</cols>
  <dt>d</dt>
  <data>
    -3.0555798992700906e-02 0. 2.4124468455964283e-03
    5.1681504633348035e-03 0.</data></distortion_coefficients>
<avg_reprojection_error>3.6778428836778632e-01</avg_reprojection_error>

Hybrid method

<camera_matrix type_id="opencv-matrix">
  <rows>3</rows>
  <cols>3</cols>
  <dt>d</dt>
  <data>
    3.0672888652406186e+03 0. 5.4174166778823223e+02 0.
    3.0724823332425444e+03 5.3686955699567227e+02 0. 0. 1.</data></camera_matrix>
<distortion_coefficients type_id="opencv-matrix">
  <rows>5</rows>
  <cols>1</cols>
  <dt>d</dt>
  <data>
    -1.0288404944496776e-01 0. -1.4505726936981191e-04
    -6.1541237963353961e-03 0.</data></distortion_coefficients>
<avg_reprojection_error>2.9886574412779127e-01</avg_reprojection_error>

[xoox]

Dr. Strobl's comments on the original test dataset:

I looked at your data and the reason probably is that your images have been taken "the OpenCV/Bouguet way." Instead, you should take more tilted images, avoiding perpendicular images (see https://www.robotic.dlr.de/fileadmin/robotic/stroblk/publications/strobl_2009iros_.pdf, page 311, bottom) and, if the software allows (DLR CalDe and DLR CalLab does), fill the image with corners (i.e., leave some corners outside the field of view). See our invaluable calibration hints in: https://www.dlr.de/rm/en/desktopdefault.aspx/tabid-3925/6084_read-9196/ Good calibration images look like this: (Older) sample images can be found here: https://www.dlr.de/rm/en/Portaldata/52/Resources/Software/CalLab/sample.zip

Using these images it's difficult if not impossible to triangulate and recover the 3D geometry of the pattern, which is precisely what my algorithm is intended to. But not only my algorithm: the basic camera calibration method is also intended at separating the effects of the position of the camera and of the focal length. This is only possible through perspective distortion or external measurements (which we don't like). Hence any calibration with that kind of images may deliver a low RMS but the parameters estimated will be wrong.

Dr. Strobl's answer to the following question.

I've a long-standing unanswered question. If we use tilted images with large oblique angles, some parts of the image will be out of focus and hence become blurry. That might decrease the corner features detection accuracy, is it right? Is my suspicion unnecessary? Or do we have to get a balance between image defocus and oblique angle?

The problem you mention is only noticeable if your camera aperture is very big or the distance to the pattern is too small. We rarely have that problem -- I even suggest to take obliques images at two different distances. Even though calibration should not pose constraints in the cameras setup, consider:

1. printing a bigger pattern and focusing at a more distant region
2. lowering the aperture size

If you still have that problem, then you're right, you should get a balance between defocus and perspective distortion (oblique images). Note that CalDe does an awesome job at detecting corners even if the images are blurry.

Did you understand why oblique images are better? Else you cannot tell range from focal length!

The problem with oblique images is that the 2-D density of corners is higher in one side of the image than in the other, hence one region of the image gets better calibrated than the other. To cope with that problem you can take 4 images with obliques images up-down-right-left and get some symmetry.

[xoox]

Here is a dataset with 9 images. Feature points were detected using CalDe. In order to compare the results with our implementation, each image has the same number of features.

The link to the dataset. dataset5.zip

Both CalLab standard method and OpenCV gave similar results.

CalLab standard method:

camera.0.k1= -0.0634703
camera.0.p1= -0.00471115
camera.0.p2= 0.00448202
camera.0.A=[ 3089.14 0.000000 663.435; 0.000000 3099.48 480.222; 0.000000 0.000000 1.00000]
camera.0.rmsInt= 0.755889

OpenCV standard method:

<camera_matrix type_id="opencv-matrix">
  <rows>3</rows>
  <cols>3</cols>
  <dt>d</dt>
  <data>
    3.0891256444690321e+03 0. 6.6352137323491718e+02 0.
    3.0994725315433725e+03 4.8023634412293796e+02 0. 0. 1.</data></camera_matrix>
<distortion_coefficients type_id="opencv-matrix">
  <rows>5</rows>
  <cols>1</cols>
  <dt>d</dt>
  <data>
    -6.3476243923252254e-02 0. -4.7097377006577604e-03
    4.4901135146666543e-03 0.</data></distortion_coefficients>
<avg_reprojection_error>7.5588754387089074e-01</avg_reprojection_error>

CalLab with refining full object structure:

camera.0.k1= -0.0899712
camera.0.p1= -0.00169304
camera.0.p2= -0.000343253
camera.0.A=[ 3056.29 0.000000 596.089; 0.000000 3037.15 439.422; 0.000000 0.000000 1.00000]
camera.0.rmsInt= 0.151591

Our implementation without initial intrinsic parameters guess:

<camera_matrix type_id="opencv-matrix">
  <rows>3</rows>
  <cols>3</cols>
  <dt>d</dt>
  <data>
    3.0936665220239684e+03 0. 6.3993195978817482e+02 0.
    3.0998850523236238e+03 5.1212558075310881e+02 0. 0. 1.</data></camera_matrix>
<distortion_coefficients type_id="opencv-matrix">
  <rows>5</rows>
  <cols>1</cols>
  <dt>d</dt>
  <data>
    -6.9046083676842063e-02 0. -2.3558689963820191e-03
    1.4073143102732205e-03 0.</data></distortion_coefficients>
<avg_reprojection_error>6.8544169962641910e-01</avg_reprojection_error>

Our implementation using OpenCV standard method as initial intrinsic parameters guess:

<camera_matrix type_id="opencv-matrix">
  <rows>3</rows>
  <cols>3</cols>
  <dt>d</dt>
  <data>
    3.0894945313852149e+03 0. 6.6349963937561552e+02 0.
    3.0974661465090298e+03 4.8023370212223756e+02 0. 0. 1.</data></camera_matrix>
<distortion_coefficients type_id="opencv-matrix">
  <rows>5</rows>
  <cols>1</cols>
  <dt>d</dt>
  <data>
    -6.5867947154494330e-02 0. -4.2237024007513498e-03
    4.9095414646612217e-03 0.</data></distortion_coefficients>
<avg_reprojection_error>4.3044986222261095e-01</avg_reprojection_error>

Our implementation did not generate the optimized results as CalLab. Further improvements or fix are needed.

We also found that different fix 3D points (x1, x2, x3. See Dr. Strobl's paper, section 3.4) will result in much different results with divers RMS and camera parameters.

The camera matrix reported by CalLab with refining full object structure was also so much different than that reported by standard method.

[xoox]

A bug was fixed in 6da25267. Our implementation reached the same RMS level as CalLab. With the same dataset of above dataset5.zip, we got the following results.

CalLab with refining full object structure:

camera.0.k1= -0.0984749
camera.0.k2= 0.127567
camera.0.A=[ 3056.33 0.000000 584.655; 0.000000 3037.10 466.268; 0.000000 0.000000 1.00000]
camera.0.rmsInt= 0.151942

Our implementation:

<camera_matrix type_id="opencv-matrix">
  <rows>3</rows>
  <cols>3</cols>
  <dt>d</dt>
  <data>
    3.0573577737911560e+03 0. 5.9777231463549515e+02 0.
    3.0362037824002300e+03 4.5311355407991232e+02 0. 0. 1.</data></camera_matrix>
<distortion_coefficients type_id="opencv-matrix">
  <rows>5</rows>
  <cols>1</cols>
  <dt>d</dt>
  <data>
    -1.0343613204866929e-01 1.7854204424225634e-01 0. 0. 0.</data></distortion_coefficients>
<avg_reprojection_error>1.5178474259880476e-01</avg_reprojection_error>

CalLab with refining full object structure:

camera.0.k1= -0.0900870
camera.0.A=[ 3055.39 0.000000 582.201; 0.000000 3036.52 459.132; 0.000000 0.000000 1.00000]
camera.0.rmsInt= 0.152238

Our implementation:

<camera_matrix type_id="opencv-matrix">
  <rows>3</rows>
  <cols>3</cols>
  <dt>d</dt>
  <data>
    3.0555752306570489e+03 0. 5.8751317251297030e+02 0.
    3.0365382066513448e+03 4.6195508556785745e+02 0. 0. 1.</data></camera_matrix>
<distortion_coefficients type_id="opencv-matrix">
  <rows>5</rows>
  <cols>1</cols>
  <dt>d</dt>
  <data>
    -8.9847714399800147e-02 0. 0. 0. 0.</data></distortion_coefficients>
<avg_reprojection_error>1.5216911603787708e-01</avg_reprojection_error>

Our implementation with feature points detected with OpenCV:

<camera_matrix type_id="opencv-matrix">
  <rows>3</rows>
  <cols>3</cols>
  <dt>d</dt>
  <data>
    3.0557118442612523e+03 0. 5.8771714504295210e+02 0.
    3.0365422748810047e+03 4.6445492814159769e+02 0. 0. 1.</data></camera_matrix>
<distortion_coefficients type_id="opencv-matrix">
  <rows>5</rows>
  <cols>1</cols>
  <dt>d</dt>
  <data>
    -8.9434361134088730e-02 0. 0. 0. 0.</data></distortion_coefficients>
<avg_reprojection_error>1.5527737807096595e-01</avg_reprojection_error>

The minor differences between camera intrinsic parameters might be the effects of choosing 3D fix points of (0, 0, 0), (d, 0, 0) and (x3, y3, 0). In our implementation these three 3D points were selected as top-left, top-right and bottom-right points of the chessboard corners grid.

[xoox]

We investigated why there are obvious differences between our results and CalLab's results as shown below.

parameter	CalLab	calibrel
fx	3055.39	3055.58
fy	3036.52	3036.54
cx	582.201	587.513
cy	459.132	461.955
k1	-0.090087	-0.0898477
RMS	0.152238	0.152169

As theoretically expected, testing has proved that the selection of fixed 3D points don't affect the result.

If we feed CalLab's results into calibrel as initial values and fix them, then a similar RMS of 0.152226 will be gotten.

If cx = 587.513 and cy = 461.955 are fixed in CalLab with values from calibrel, CalLab will reach even smaller RMS of 0.152181 and give out the following parameters which are closer to calibrel's calculation.

parameter	CalLab
fx	3055.57
fy	3036.54
k1	-0.0899015

We can get a conclusion that CalLab fails to reach better optimization results especially for cx and cy. The possible reasons are following.

• Jacobian is numerically approximated in CalLab while it's computed using analytical expressions in calibrel.
• Single precision float numbers are used internally in CalLab instead of double precision?
• Each image point's derivative of cx or cy is always 1 or 0, so the optimized values of cx and cy are more sensitive to other parameters' calculation or round errors.

But the above guesses can't explain CalLab gives same results as OpenCV when refining full object structure is not requested. Further detailed investigations are still needed.

CalLab is very sensitive to Levenberg-Marquardt parameters. Here is a comparison.

parameter	FTOL=0.0001, RELSTEP=0.01	FTOL=0.00001, RELSTEP=0.045
fx	3055.39	3055.66
fy	3036.52	3036.42
cx	582.201	588.023
cy	459.132	461.368
k1	-0.090087	0.0899177
RMS	0.152238	0.152171

[xoox]

CalLab and calibrel were compared again with dataset1, dataset2 and dataset3 in the follow archive.

https://github.com/xoox/calibrel_testdata/archive/CalLab.zip

The three above dataset were sampled from the same dataset actually which was split into three parts. So camera parameters calculated from them should be consistent to each other. The feature points were detected with OpenCV and refined with cornerSubPix().

CalLab results

FTOL=0.00001. Other optimization parameters were set to default. RELSTEP=0.01

Parameter	dataset1	dataset2	dataset3
RMS	0.0867361	0.0895668	0.0861593
fx	3040.94	3040.96	3040.30
fy	3041.17	3040.16	3038.64
cx	607.857	608.843	607.461
cy	536.170	537.148	537.625
k1	-0.0829592	-0.0822956	-0.0837340

For RELSTEP=0.04

Parameter	dataset1	dataset2	dataset3
RMS	0.0867251	0.0895250	0.0861507
fx	3041.04	3040.99	3040.38
fy	3041.11	3040.19	3038.73
cx	608.782	607.429	607.820
cy	537.467	537.160	538.726
k1	-0.0829699	-0.0823135	-0.0837408

For RELSTEP=0.0025

Parameter	dataset1	dataset2	dataset3
RMS	0.0869889	0.0896314	0.0864126
fx	3040.74	3040.98	3040.14
fy	3041.16	3040.17	3038.54
cx	609.737	609.443	612.475
cy	533.515	536.100	533.012
k1	-0.0828763	-0.0822277	-0.0837854

When RELSTEP=0.04 was used, CalLab's results were closer to calibrel's results.

calibrel results

Parameter	dataset1	dataset2	dataset3
RMS	0.0867244	0.0895241	0.0861504
fx	3041.01665	3040.99082	3040.37711
fy	3041.10761	3040.19260	3038.72519
cx	608.810470	607.321286	607.935824
cy	537.219851	537.245663	538.764664
k1	-0.0829604	-0.0823206	-0.0837447

CalLab's sensitivity to RELSTEP

With dataset5.zip, FTOL=0.00001, GTOL=1e-6. For standard calibration.

Parameter	RELSTEP=0.1	RELSTEP=0.01	RELSTEP=0.001
RMS	0.798080	0.798078	0.798078
fx	3089.99	3089.82	3089.81
fy	3100.11	3100.00	3099.99
cx	619.898	619.939	619.944
cy	516.445	516.496	516.499
k1	-0.0639205	-0.0639682	-0.0639729

The standard calibration method is stable with regard to RELSTEP.

With the same dataset5, if refining full object structure, the results will diverge from each other (especially cx and cy).

Parameter	RELSTEP=0.1	RELSTEP=0.01	RELSTEP=0.001
RMS	0.152505	0.152237	0.156158
fx	3056.14	3055.39	3056.14
fy	3034.46	3036.52	3037.84
cx	600.033	582.201	611.249
cy	436.028	459.132	513.135
k1	-0.0918021	-0.0900870	-0.0866259

When the dimensionality of the parameter space is much higher, cx and cy are perhaps more sensitive to jacobian. This kind of diversity was also found upon dataset1, dataset2 and dataset3, but the differences were smaller. dataset5 might has not full coverage of field of view. And the plate used in dataset5 was not as rigid as the one in dataset1, dataset2 and dataset3.