Multi-camera non-overlapping calibration output is unreasonable

[DonovanZhu]

System information (version)

• Operating System / Platform => Ubuntu 20.04
• OpenCV => 4.5.5
• Ceres => 2.2.0
• Boost => 1.71.0
• C++ => 17
• Compiler => cmake 3.24,2

Vision system

• Number of cameras => 4
• Cameras type => Realsense D435 (only use left IR and close emitter)
• Types of cameras => perspective
• Multicamera configurations => non-overlapping
• Configuration file => *.yml
• Image sequences => 179 images per camera
• Image size => 848 x 480
• Hardware synchronization => Yes
• Charuco boards number => 4
• Charuco boards size => all are 100mm square size of each grid
• Charuco boards index => [0, 1, 2, 3]
• Tried as many different viewpointas as possible during calibration

One of the image is like this:

The calib_param.yml file is as follow: calib_param_Non-overlapping.txt

Problem:

A list of images taken by each camera. We could call the four lists A/B/C/D. At first, I tried calibrate all of them together. However, the output makes no sense. As you can see, the intrinsics are too small and the extrinsics are too big:

calibrated_cameras_data.txt reprojection_error_data.txt

The strange is, I get rid of the image list "D", and only calibrate images "A/B/C". Without other changes (even same images as before), the output then seems good:

calibrated_cameras_data.txt reprojection_error_data.txt

Then I try only use images lists "C/D", the results are reasonable:

calibrated_cameras_data.txt reprojection_error_data.txt

Finally, I set fix_intrinsic to "1", and calibrate all the four cameras together. The outputs are also not good. And it changes a lot every time I rerun the program again. reprojection_error_data.txt calibrated_cameras_data.txt

Why this happens? How do I fix this problem. Thanks!

[DonovanZhu]

I found that before non-overlapping handeye calibration step, the reprojection errors are still small. But after that step, the calibration result seems unreasonable. Then I change the bootstrapped he technique to traditional method, the problems are solved and the final output is normal right now.

Is there any suggestions for using the bootstrapped he technique?

[rameau-fr]

Hello, Thank you very much for using MC-Calib and for your detailed feedback! It's great to hear that you were able to identify the cause of your problem; for now, I would like to suggest that you try using MC-Calib without the bootstrapped HE strategy and instead rely on the normal technique, as everything is refined non-linearly afterwards. Initially, we developed the bootstrapped technique to improve the speed and robustness. We extensively tested this approach, but it seems that your scenario is causing difficulties with the bootstrapped technique. Without analyzing your data, it's hard to pinpoint the exact problem. However, I would be happy to take a closer look if you would be willing to share your data with me.

Thank you again for your feedback. We greatly appreciate it, as it helps us improve this toolbox.

[zjp-tju]

fickbob