Unreasonable calibration output

[jhk0219]

System information (version)

• Operating System / Platform => ❔
• OpenCV => ❔
• Ceres => ❔
• Boost => ❔
• C++ => ❔
• Compiler => ❔

Vision system

• Number of cameras => 2
• Types of cameras => perspective (perspective, fisheye, hybrid)
• Multicamera configurations => overlapping (overlapping, non-overlapping, converging)
• Configuration file => ❔ (i.e. *.yml)
• Image sequences => ❔
  • number of images per camera
  • if relevant and possible, please share image sequences

Describe the issue / bug

Please provide a clear and concise description of what the issue / bug is.

Hello,

We are attempting to calibrate two cameras (eventually, 5) for proof of concept using a printed charuco board on a 24 inch x 36 inch poster board. We printed this using 4000x6000 resolution at a neighborhood printing center. We have one camera with 1920x1080 resolution, and another with 2048x732. To make this easier, I will call the1920x1080 "Camera 1" and the 2048x732 "Camera 2".

Camera 1 has the ability to zoom in/out and focus but Camera 2 does not. We currently have a stream running of Camera 1, from which snap photos every ~8seconds. We found that Camera 2 has issues with quality when taking photos from a stream, so we are pulling them directly from the camera and encoding into a BMP. We take these photos every ~9 seconds.

We've tried a variety of approaches to gathering the data for the calibration, but here are the constants:

1. We point both cameras to a specific spot, which as much overlap as possible. Since Camera1 has the ability to zoom. we often find that Camera 2's frame has a longer length. In terms of width, both frames are similar. These cameras are mounted on top of a vehicle.
2. We have the 24inch x 36inch board mounted/taped to a dolly/hand cart. Since we are taking these pictures outside, we place the back of the board against the sun as to reduce possibility of glare.
3. We move the board around on the camera frames, trying to cover every inch of the frame on both cameras. There are instance where none/small parts of the board is shown on Camera 1, but is still visible on Camera 2. We leave the board still for about 30 seconds, allowing us to take 3-4 photos per position. We then move the dolly/hand cart to find a new position.

We originally took about ~300 photos on each camera ~60 meters away from the cameras. We moved the board around and tried to fill every spot on the frame, while trying to leave at 50% of the board in the frame. Since the pictures are taken every X seconds, there were photos were the board was in transit/turned/not fully secured on the spot we wanted to put it. Upon running MC-Calib, we ran into the error code :

what(): OpenCV(4.2.0) ../modules/core/src/matrix_expressions.cpp:23: error: (-5:Bad argument) Matrix operand is an empty matrix. in function 'checkOperandsExist'

We then thought perhaps the board is too far away and it is harder to see the squares for the camera at that distance. We wanted proof of concept that the program could work with closer pictures. We moved the board to about ~15m from the cameras, which resulted in the board almost covering the entirety of the Y axis on Camera 2. We adjusted Camera 1's zoom to fit, and took about 25 photos on each camera, once again taking pictures every 8-9 seconds from the stream. We took most of the pictures towards the center of the frame. We were able to run the program without an error, but got wildly unexpected values for the respective positions of the cameras. Our rough measurements with a tape measure says the cameras are about 40 inches x 40 inches x 0 inches away from each other. The program seemed to suggest the cameras are 150-200 inches away in each direction.

We proceeded to take pictures at the same distance (15 meters). We grabbed about 800 photos on each camera, moving the board to various positions around the frames. Once again, we ran into the same error code as above.

We then moved the board to about 18-20 meters where we felt Camera#2 had the best focus. We moved the board up and down the frame over the course of a few hours, taking ~1200 photos on each camera. Same error as above.

We feel that we're running out of options. It's difficult for us to get perfect quality photos since the photos are being taken outside, not in a controlled environment.

Our questions are:

1. Is it important to delete/omit photos in which the board is in transit or not in full view ? Since we are taking photos every X seconds, there will be times where the board is in transit/slanted and does not give full view. The cameras are not perfectly synced in when they take the photo, so some of the photos will have discrepancies in the board position.
2. Does every picture need to show the board in both of the cameras' frames? Since one of the cameras cover a wider width than the other, we do have pictures where the board exists on one, but not the other.

If you have any recommendations on how we can improve the way we take data, I would really appreciate it.

Thank you

[rameau-fr]

Thank you very much for your interest in this toolbox.

I have a few questions regarding your setup:

• Zoom/Focus: Did you fixed the focus and zoom of Camera 1? A camera with auto-focus and variable focal length cannot be calibrated this way.
• Individual calibration: Have you tried calibrating Camera 1 and Camera 2 separately? What happens when you run the code with a single camera?
• Versioning: Are you using the Docker provided with MC-Calib to avoid any versioning issues?
• Syncrho: Are the cameras perfectly synchronized? If you have motion blur, I suspect something is moving (either in the scene or due to camera ego-motion). If the cameras are not synchronized via any hardware, then the board and the cameras should be perfectly static during the image capture.
• log: Could you provide us with the entire log? It would help to see exactly where the code is crashing.
• Config: Could you also share your configuration file?
• Shutter: Are your cameras CCD or CMOS? (rolling or global shutter)
• Moving elements: Do you move the board or the cart?

A few clarifications:

• Motion blur: Normally, images with excessive motion blur should be automatically rejected, either because the Aruco markers cannot be detected, or when the points are refined, or via RANSAC (depending on the threshold and amount of blur). However, it is certainly better to input proper pictures in the first place.
• It is completely okay if the board is partly visible on one or both cameras, but you might need to adjust the parameter min_perc_pts to a lower value (e.g., 0.3 --> 30% of the corner visible required to consider the board for calibration)
• The board's distance should not necessarily be a major issue (it should still be possible to calibrate with a good reprojection error).
• If you are calibrating cameras with significantly different focal lengths, you might want to print more than one checkerboard, for instance, a large one and a smaller one.
• Overlap is not strictly necessary for MC-Calib, but it would greatly facilitate the calibration. Moreover, knowing your cameras are mounted on a platform without Y-axis freedom, you might encounter severe degeneracy problems if you rely solely on the non-overlapping capability of the toolbox.

I am looking forward to hearing back from you and seeing how we can help you. Also, please do not hesitate to share some of your pictures with us if possible.

[jhk0219]

Thank you for your response. We have tried some new things but have not yet achieved calibration. Instead of trying to calibrate two different cameras, we chose to calibrate two of the same. We have one camera with focus and zoom capabilities, and four cameras without those capabilities. We tried to calibrate two of the four cameras, to see if that would bring better results.

I have a few questions regarding your setup:

- Zoom/Focus: Did you fixed the focus and zoom of Camera 1? A camera with auto-focus and variable focal length cannot be calibrated this way. +Camera 1 has the ability to zoom and focus. Camera 2 does not have either. We turned off auto focus on camera 1 and moved the frames on both cameras to overlap as much as possible. Because Camera2 has a much wider frame, there are parts of the frame where camera 1 cannot see

- Individual calibration: Have you tried calibrating Camera 1 and Camera 2 separately? What happens when you run the code with a single camera? +Yes, we were not able to get good results. We got infinite error values

- Versioning: Are you using the Docker provided with MC-Calib to avoid any versioning issues? +Yes we installed using MC-Calib's instructions

- Syncrho: Are the cameras perfectly synchronized? If you have motion blur, I suspect something is moving (either in the scene or due to camera ego-motion). If the cameras are not synchronized via any hardware, then the board and the cameras should be perfectly static during the image capture. +Hardware synchronization is impossible for out setup. Instead, we did notice very small amounts of movement on the camera, either due to wind or other factors. We tried to create an environment where nothing in the frame moves.

- log: Could you provide us with the entire log? It would help to see exactly where the code is crashing. - Config: Could you also share your configuration file? %YAML:1.0

config for calibration of drvid cameras

######################################## Boards Parameters ###################################################

number_board: 1 # number of boards used for calibration (for overlapping camera 1 is enough ...)

number_x_square: 4 #number of squares in the X direction

number_y_square: 6 ##number of squares in the Y direction

resolution_x: 4000  # horizontal resolution in pixel

resolution_y: 6000  # vertical resolution in pixel

length_square: 0.04  # parameters on the marker (can be kept as it is)

length_marker: 0.03 # parameters on the marker (can be kept as it is)

number_board: 1 # number of boards used for calibration (for overlapping camera 1 is enough ...)

square_size: 5.4375 #5.4375

cam_prefix: "Cam_"

root_path: "../data/"

######################################## Camera Parameters ###################################################

distortion_model: 0 #0:Brown (perspective) // 1: Kannala (fisheye)

distortion_per_camera : []

number_camera: 2  # number of cameras in the rig to calibrate

refine_corner: 1  # activate or deactivate the corner refinement

min_perc_pts: 0.7  # min percentage of points visible to assume a good detection

# cams_params_path: "../intrinsic/double-drvid-curated.yml"

# fix_intrinsic: 1

######################################## Optimization Parameters ###################################################

quaternion_averaging: 1    # use Quaternion Averaging or median for average rotation

ransac_threshold: 3        # RANSAC threshold in pixel (keep it high just to remove strong outliers)

number_iterations: 1000    # Max number of iterations for the non linear refinement

######################################## Hand-eye method #############################################

he_approach: 0 #0: bootstrapped he technique, 1: traditional he

######################################## Output Parameters ###################################################

save_path: "workthistime/"

save_detection: 1

save_reprojection: 1

camera_params_file_name: ""

- Shutter: Are your cameras CCD or CMOS? (rolling or global shutter) - Moving elements: Do you move the board or the cart? +We moved the cart after pictures were taken from the stream, as to cover the entire frame of the camera.

Following are the pictures we used for the calibration. The Drv1 and drv2 folders are pictures from our last attempt. In this attempt, we used two cameras that have 2048x732 resolution. Neither of these cameras have focus or zoom capabilities. The error message and configuration parameters above are from this attempt.

https://drive.google.com/drive/folders/1smcpgeDjNZrd_Y_Vu1MqarU9XAFN72-j?usp=sharing

[rameau-fr]

Thank you very much for sharing all this information with us. I have looked at the pictures you shared, and I might have some comments:

• Certain parameters seem to be missing in your configuration file such as number_x_square_per_board and other, I am not sure if it can cause some issue
• From your log it seems the toolbox struggles to detect the keypoints in the board properly (it only detects one or 3 points maximum which is not sufficient to calibrate the camera). Try to set the min_perc_pts to 0.5 to ensure you have sufficient points to calibrate (even if 0.7 should also work actually). If the problem of detection persists, could you please give a try with a square board (just in case it is a bug in the toolbox for a rectangular shape)?
• The number of images for each camera should be exactly the same to perform multi-camera calibration. Even if the image captured with one camera does not contain the checkerboard, it should be included in the folder.
• The naming of the images should ideally include proper zero padding, such as cam01_00001.jpeg, cam01_00002.jpeg, etc.
• The images you use lack diversity; the board always has the same orientation. You should try to diversify the viewpoints by tilting and panning the board a bit, or it can affect the calibration accuracy.

What I recommend doing here is calibrating your system in two stages. First, calibrate the intrinsics of both cameras individually, which will give you more freedom to acquire proper images. Then, you can use these intrinsic parameters and calibrate only the extrinsic parameters using MC-Calib.

I hope it is helping. Best of luck with your calibration, and do not hesitate to seek help here!

[jhk0219]

Based on the pictures from the drive link I sent you, did you feel like image quality was an issue? We were wondering if the images we took had good enough quality for the toolbox

[rameau-fr]

My sincere apologies for getting back to you so late. I have been quite busy in the past month due to other responsibilities. In my view, the quality seems acceptable. However, have you tried to implement all the feedback I mentioned in my previous message? I believe the naming convention and the number of images are definitely problematic in your case. Please remember that the images should have consistent naming, and even if the checkerboard is not visible, all the images captured by the cameras should be provided. I hope you have managed to address these issues. In my opinion, if you can resolve the problems I mentioned in my previous message, everything should work properly!