Hand-Eye Calibration Assertion Failure with Non-Overlapping 4-Camera Fisheye Setup

[EladDan]

System information (version)

• Operating System / Platform => Ubuntu 20.04.6 LTS + your docker
• mc-calib version/commit => 62d0587 (master branch)

Vision system

• Number of cameras => 4

• Types of cameras => fisheye (160 deg)

• Multicamera configurations => non-overlapping

• Configuration file => config link

• Image sequences => images link

  • number of images per camera: ~30

• Description: My setup involves 4 non-overlapping fisheye cameras, using 2 groups of charuco boards. Each group contains 2 boards, always visible to 2 cameras. After every ~30 images, the cameras system is rotated, making the boards seen by other 2 cameras.

Sketch of the cameras (1-4) + boards (A-D) setup

Describe the issue / bug

During the calibration process, immediately after the "Non-overlapping calibration initiated" step, the following error occurs: OpenCV(4.2.0) ../modules/calib3d/src/calibration_handeye.cpp:705: error: (-215:Assertion failed) Rgripper2base.size() >= 3 in function 'calibrateHandEye'

Console Log

I would greatly appreciate any guidance on resolving this failure. Thank you for your ongoing support and development of MC-Calib.

[rameau-fr]

Thank you very much for using MC-Calib and for your feedback. You have identified a very interesting corner case, which, fortunately, was quite easy to debug. The error occurs when initializing the camera groups. Our graph-based strategy discovers that 2 cameras are related, but no direct poses are yet known, leading to initialization failure.

This can easily be resolved by adding the following piece of code:

// Check if enough common poses are available:
if (pose_abs_1.size() <= 3 || pose_abs_2.size() <= 3) { return; }

to line 1448 of McCalib.cpp. For reference, see the code on GitHub: McCalib.cpp Line 1448. Then, you should be able to complete your calibration successfully. I achieved a mean reprojection error of 0.174003px, which is quite good. (Do not forget to compile again before testing.) This is just a quick fix, but I will discuss it more with @BAILOOL to possibly come up with something more elegant.

I also have some additional comments:

• The motion you are using seems very close to being degenerate. You have almost no tilt rotation nor translation on the y-axis. Therefore, even if you achieve a good reprojection error, it does not necessarily mean that the translation on the y-axis is well estimated. I recommend acquiring your image set with a more diverse range of motions.
• In the configuration file, resolution_x and resolution_y refer not to the resolution of the image but to the resolution of the calibration pattern you printed. It should not cause any issue, but I prefer to specify it in case.
• Also, the coverage of your images is quite limited, which can cause some inaccuracy in the intrinsic parameters of the cameras (especially for fisheye cameras). Try to cover the entire sensors or/and use more boards, diversify the motion. Otherwise, you can also pre-calibrate the intrinsic parameters of the cameras if needed.

Please let me know if everything works for you.

[EladDan]

I've successfully calibrated the system using the fix you suggested, and it worked perfectly! Thank you so much for your quick and effective solution.

Also, I greatly appreciate your additional comments and insights. The images we used were initially taken just to validate that the system could be calibrated in this configuration. Rest assured, we will implement your suggestions for a more diverse range of motions and better image coverage to improve our calibration accuracy.

Thanks again for your support and guidance.