Camera Rotation&Translation Matrix in rot_trans_cX.dat file

[valioiv]

Hi, sorry for asking here again but I don't get what to put inside rot_trans_cX.dat configuration file especially for "R" and T:

For "R" I can see there 3 lines of 3 coefficients each. But I've run the calibration script ( https://temugeb.github.io/opencv/python/2021/02/02/stereo-camera-calibration-and-triangulation.html )on 9 images from each one of the two cameras and this is the output for the left camera:

Same as text:

rmse: 1.6845785933692061
camera matrix:
 [[334.79496397   0.         325.91766479]
 [  0.         338.90436734 271.32265213]
 [  0.           0.           1.        ]]
distortion coeffs: [[-0.34544306  0.16894141  0.00139881 -0.00862738 -0.0428115 ]]
Rs:
 (array([[-0.16279281],
       [ 0.0692457 ],
       [ 1.59570888]]), array([[-0.67354443],
       [ 0.49915959],
       [ 1.56042775]]), array([[ 0.36843621],
       [-0.4983611 ],
       [ 1.58666566]]), array([[0.48836489],
       [0.53081859],
       [1.5679168 ]]), array([[ 0.34228534],
       [ 0.4557069 ],
       [-1.51804147]]), array([[-0.02492577],
       [ 0.02677482],
       [ 1.63894725]]), array([[0.1350394 ],
       [0.34949505],
       [1.64581647]]), array([[-0.85483454],
       [-0.47871847],
       [ 1.58409214]]), array([[-1.06467574],
       [ 1.0698896 ],
       [ 1.43298644]]))
Ts:
 (array([[ 7.23488744],
       [-0.44975245],
       [ 9.82438543]]), array([[ 7.38132806],
       [ 1.70549895],
       [13.49581392]]), array([[ 8.6187057 ],
       [-0.19591065],
       [ 8.25102323]]), array([[14.05962215],
       [ 0.51148509],
       [14.77727402]]), array([[-0.09701999],
       [ 2.35128648],
       [21.41633449]]), array([[ 7.00303687],
       [ 0.9519228 ],
       [22.26382956]]), array([[11.24380084],
       [-0.21628372],
       [ 9.77399498]]), array([[-1.25008991],
       [ 1.22434956],
       [19.64095728]]), array([[ 7.7751413 ],
       [ 6.9388801 ],
       [22.33717076]]))

Do I need to put in the rot_trans_cX.dat file all the 9 arrays like that or I need to put just some random 3 of them? This is what I currently have (I've selected random 3 of them):

R:
-0.16279281 0.0692457 1.59570888
-0.67354443 0.49915959 1.56042775
0.36843621 -0.4983611 1.58666566
T:
7.23488744
-0.44975245
9.82438543

Is it correct? What about "T" ?

[TemugeB]

Hello,

Are the nine R and T's returned from cv.calibrateCamera or cv.stereoCalibrate ? It looks like it is from cv.calibrateCamera. If it is, then in my blog post, we discard the R and T values from this step and use the camera matrix in the cv.stereoCalibrate call. Then cv.stereoCalibrate returns R and T that takes points in camera0 space to camera1 space.

If you put in any of the R and T pairs you obtained from your 9 images into rot_trans_c0.dat, then you are picking one of the images, or rather the position of the calibration pattern, as your worldspace origin point. That means all 3D points obtained from triangulation will be with respect to this origin point defined by the calibration pattern. In any case, this is what you need to do to get working calibration matrices:

World space origin defined by calibration pattern

1. Chose one of the R and T values obtained from cv.calibrateCamera in rot_trans_c0.dat. We will call these R0 and T0.
2. Use cv.stereoCalibrate to obtain R and T that performs coordinate transform from cam0 to cam1. We will call these R01 and T01 to indicate these are used to translate points from cam0 coordinate frame to cam1 coordinate frame.
3. Directly calculate R and T that can take world space points and return cam1 coordinates. We will call these R1 and T1. These can be calculated as: R1 = R01 @ R1 T1 = R01 @ T0 + T01 (@ means matrix multiply) Then put R1 and T1 into rot_trans_c1.dat

In my blog post, I simply select R0 as identity matrix and T0 as zeros vector. This essentially chooses world space origin to overlap with cam0 coordinates.

On a side note, you have an rmse of 1.6. This is too high in my opinion. You should aim to obtain less than 1. A very good calibration should be less then 0.2 and acceptable level will be about 0.5. The calibration pattern should be printed on a large sheet and it should be as flat as you can make it. The calibration steps are very sensitive to small distortions in the calibration pattern.

[valioiv]

@TemugeB , I'm so sorry for the delay. Now I can confirm that based on your recommendations I managed to make proper calibration! I just put world space origin to match with cam0 coordinates and it works perfectly fine! The main problem was that I had bad (and missing) images during the image gathering before the actual calibration and this was the reason for the bad RMSE and many other matrix issues. Redone the capturing of the images, recalibrate and now everything is up and running great!