Kalibr Imu-Camera calibration failure on example dataset

[GabiShahmayster]

@AbanobSoliman Hello, First of all, thank you very much for providing this data. It's an amazing work and what's even more amazing is the extent of your attempt to make the data public and reproducible. Thank you very much.

I've tried running the camera(s)-to-imu calibration using Kalibr and ROS bag and YAML files you've provided. Unfortunately the optimization failed. I was wondering if you could spot any issues with the config/data/kalibr input I've used?

Config/Data

1. Checkerboard: sharepoint/Datasets/yaml_config_files/calibration/checkerboard.yaml
2. Cameras: sharepoint/Datasets/yaml_config_files/calibration/rgb_pinhole_checkerboard/radtan/camchain-SVI.yaml
3. IMU: sharepoint/Datasets/yaml_config_files/calibration/imu0.yaml
4. ROS bag: sharepoint/Datasets/Calibration/Checkerboard/SVI/SVI.bag

Kalibr input: rosrun kalibr kalibr_calibrate_imu_camera --bag SVI.bag --cam camchain-SVI.yaml --imu imu0.yaml --target checkerboard.yaml --verbose

Kalibr consile output: Unfortunately, the optimization failed: Extracted corners for 991 images (of 2192 images)
Baseline between cam0 and cam1 set to: T= [[ 0.99999996 0.00007783 -0.00028918 -0.40306714] [-0.00007903 0.99999139 -0.00414943 0.01028023] [ 0.00028885 0.00414946 0.99999135 -0.00365584] [ 0. 0. 0. 1. ]] Baseline: 0.403214794416 [m]

Building the problem Spline order: 6 Pose knots per second: 100 Do pose motion regularization: False xddot translation variance: 1000000.000000 xddot rotation variance: 100000.000000 Bias knots per second: 50 Do bias motion regularization: True Blake-Zisserman on reprojection errors -1 Acceleration Huber width (sigma): -1.000000 Gyroscope Huber width (sigma): -1.000000 Do time calibration: True Max iterations: 30 Time offset padding: 0.030000 Estimating time shift camera to imu:

Initializing a pose spline with 15005 knots (100.000000 knots per second over 150.049997 seconds) [DEBUG] [1678870107.831322]: discrete time shift: -11 [DEBUG] [1678870107.833658]: cont. time shift: 0.0549999987706 [DEBUG] [1678870107.835281]: dT: 0.00499999988824 Time shift camera to imu (t_imu = t_cam + shift): 0.0549999987706 Estimating time shift camera to imu:

Initializing a pose spline with 15365 knots (100.000000 knots per second over 153.649997 seconds) [DEBUG] [1678870120.001346]: discrete time shift: 9 [DEBUG] [1678870120.009414]: cont. time shift: -0.0449999989942 [DEBUG] [1678870120.012453]: dT: 0.00499999988824 Time shift camera to imu (t_imu = t_cam + shift): -0.0449999989942

Estimating imu-camera rotation prior

Initializing a pose spline with 15005 knots (100.000000 knots per second over 150.049997 seconds) Gravity was intialized to [-2.53208877 -2.38991653 9.16761956] [m/s^2] Orientation prior camera-imu found as: (T_i_c) [[ 0.97326135 0.03451553 0.22709255] [-0.06203114 0.99140744 0.11516693] [-0.2211662 -0.12617433 0.96703958]] Gyro bias prior found as: (b_gyro) [ 0.00516027 -0.02917348 0.01284601]

Initializing a pose spline with 15017 knots (100.000000 knots per second over 150.169997 seconds)

Initializing the bias splines with 7508 knots

Adding camera error terms (/cam0/image_raw) Added 1029 camera error terms

Adding camera error terms (/cam1/image_raw) Added 991 camera error terms

Adding accelerometer error terms (/imu0) Added 30035 of 31130 accelerometer error terms (skipped 1095 out-of-bounds measurements)

Adding gyroscope error terms (/imu0) Added 30035 of 31130 gyroscope error terms (skipped 1095 out-of-bounds measurements)

Before Optimization

Normalized Residuals

Reprojection error (cam0): mean 9.30784900331e+17, median 5.74944856736, std: 2.04940548709e+20 Reprojection error (cam1): mean 1.27196369565e+19, median 31.9544352382, std: 2.74259257464e+21 Gyroscope error (imu0): mean 20.6529919192, median 6.96469918624, std: 30.2779479108 Accelerometer error (imu0): mean 48.057652868, median 17.9841865566, std: 88.7268700246

Residuals

Reprojection error (cam0) [px]: mean 9.30784900331e+17, median 5.74944856736, std: 2.04940548709e+20 Reprojection error (cam1) [px]: mean 1.27196369565e+19, median 31.9544352382, std: 2.74259257464e+21 Gyroscope error (imu0) [rad/s]: mean 3.80631681012, median 1.28358407797, std: 5.58018240455 Accelerometer error (imu0) [m/s^2]: mean 75.0069385932, median 28.0691772527, std: 138.482229038

Optimizing... Using the block_cholesky linear system solver Using the levenberg_marquardt trust region policy Using the block_cholesky linear system solver Using the levenberg_marquardt trust region policy Initializing Optimization problem initialized with 30053 design variables and 155089 error terms The Jacobian matrix is 370246 x 135220 [0.0]: J: 3.52079e+47 CHOLMOD warning: matrix not positive definite. file: ../Supernodal/t_cholmod_super_numeric.c line: 911 [WARNING] System solution failed CHOLMOD warning: matrix not positive definite. file: ../Supernodal/t_cholmod_super_numeric.c line: 911 [WARNING] System solution failed CHOLMOD warning: matrix not positive definite. file: ../Supernodal/t_cholmod_super_numeric.c line: 911 [WARNING] System solution failed CHOLMOD warning: matrix not positive definite. file: ../Supernodal/t_cholmod_super_numeric.c line: 911 [WARNING] System solution failed CHOLMOD warning: matrix not positive definite. file: ../Supernodal/t_cholmod_super_numeric.c line: 911 [WARNING] System solution failed CHOLMOD warning: matrix not positive definite. file: ../Supernodal/t_cholmod_super_numeric.c line: 911 [WARNING] System solution failed CHOLMOD warning: matrix not positive definite. file: ../Supernodal/t_cholmod_super_numeric.c line: 911 [WARNING] System solution failed CHOLMOD warning: matrix not positive definite. file: ../Supernodal/t_cholmod_super_numeric.c line: 911 [WARNING] System solution failed CHOLMOD warning: matrix not positive definite. file: ../Supernodal/t_cholmod_super_numeric.c line: 911 [WARNING] System solution failed CHOLMOD warning: matrix not positive definite. file: ../Supernodal/t_cholmod_super_numeric.c line: 911 [WARNING] System solution failed CHOLMOD warning: matrix not positive definite. file: ../Supernodal/t_cholmod_super_numeric.c line: 911 [WARNING] System solution failed [1]: J: 5.68623e+46, dJ: 2.95217e+47, deltaX: 5.03379e-06, LM - lambda:1.51116e+24 mu:4096 [2]: J: 9.1835e+45, dJ: 4.76788e+46, deltaX: 6.01591e-06, LM - lambda:1.04228e+24 mu:2 [3]: J: 1.48317e+45, dJ: 7.70033e+45, deltaX: 7.16105e-06, LM - lambda:7.18884e+23 mu:2 [4]: J: 2.39538e+44, dJ: 1.24363e+45, deltaX: 8.34259e-06, LM - lambda:4.9583e+23 mu:2 [5]: J: 3.86864e+43, dJ: 2.00851e+44, deltaX: 9.0466e-06, LM - lambda:3.41985e+23 mu:2 [6]: J: 6.24806e+42, dJ: 3.24383e+43, deltaX: 8.71271e-06, LM - lambda:2.35875e+23 mu:2 [7]: J: 1.0091e+42, dJ: 5.23896e+42, deltaX: 8.5844e-06, LM - lambda:1.62689e+23 mu:2 [8]: J: 1.62977e+41, dJ: 8.46122e+41, deltaX: 9.51981e-06, LM - lambda:1.12211e+23 mu:2 [9]: J: 2.63227e+40, dJ: 1.36654e+41, deltaX: 1.09998e-05, LM - lambda:7.73943e+22 mu:2 [10]: J: 4.25187e+39, dJ: 2.20708e+40, deltaX: 1.28053e-05, LM - lambda:5.33797e+22 mu:2 [11]: J: 6.87106e+38, dJ: 3.56477e+39, deltaX: 1.49212e-05, LM - lambda:3.68141e+22 mu:2 [12]: J: 1.11273e+38, dJ: 5.75833e+38, deltaX: 1.72932e-05, LM - lambda:2.53826e+22 mu:2 [13]: J: 1.82624e+37, dJ: 9.30106e+37, deltaX: 1.94859e-05, LM - lambda:1.7483e+22 mu:2 [14]: J: 3.30013e+36, dJ: 1.49623e+37, deltaX: 1.9614e-05, LM - lambda:1.20019e+22 mu:2 [15]: J: 8.37265e+35, dJ: 2.46287e+36, deltaX: 2.52838e-05, LM - lambda:8.14739e+21 mu:2 [16]: J: 2.51214e+35, dJ: 5.8605e+35, deltaX: 4.42178e-05, LM - lambda:4.38627e+21 mu:2 [17]: J: 5.79558e+34, dJ: 1.93259e+35, deltaX: 6.84712e-05, LM - lambda:1.46209e+21 mu:2 [18]: J: 1.2474e+34, dJ: 4.54819e+34, deltaX: 8.59966e-05, LM - lambda:4.87363e+20 mu:2 [19]: J: 3.13421e+33, dJ: 9.33975e+33, deltaX: 9.93952e-05, LM - lambda:1.87216e+20 mu:2 [20]: J: 1.15631e+33, dJ: 1.9779e+33, deltaX: 0.00010614, LM - lambda:7.72978e+19 mu:2 [21]: J: 6.29764e+32, dJ: 5.26545e+32, deltaX: 0.000123554, LM - lambda:3.48547e+19 mu:2 [22]: J: 5.12488e+32, dJ: 1.17277e+32, deltaX: 0.000349816, LM - lambda:1.64248e+19 mu:2 [23]: J: 3.53631e+32, dJ: 1.58856e+32, deltaX: 0.000116293, LM - lambda:1.64261e+19 mu:2 [24]: J: 2.77639e+32, dJ: 7.59923e+31, deltaX: 0.000391204, LM - lambda:8.30045e+18 mu:2 [25]: J: 1.43006e+32, dJ: 1.34633e+32, deltaX: 0.0014354, LM - lambda:2.76682e+18 mu:2 [26]: J: 4.37255e+31, dJ: 9.928e+31, deltaX: 0.00358153, LM - lambda:9.22272e+17 mu:2 [27]: J: 1.44604e+31, dJ: 2.92652e+31, deltaX: 0.00532256, LM - lambda:3.07424e+17 mu:2 [28]: J: 5.0207e+30, dJ: 9.43965e+30, deltaX: 0.00313107, LM - lambda:2.08334e+17 mu:2 Last step was a regression. Reverting [29]: J: 1.09057e+33, dJ: -1.08555e+33, deltaX: 0.00563168, LM - lambda:6.94445e+16 mu:2 [30]: J: 3.33915e+30, dJ: 1.68155e+30, deltaX: 0.00118579, LM - lambda:2.77778e+17 mu:4 [ERROR] [1678870525.199510]: Optimization failed! Traceback (most recent call last): File "/catkin_ws/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_calibrate_imu_camera", line 247, in main() File "/catkin_ws/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_calibrate_imu_camera", line 210, in main iCal.optimize(maxIterations=parsed.max_iter, recoverCov=parsed.recover_cov) File "/catkin_ws/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccCalibrator.py", line 180, in optimize raise RuntimeError("Optimization failed!") RuntimeError: Optimization failed!

[AbanobSoliman]

Hi @GabiShahmayster ,

Thank you for highly appreciated feedback.

Sure, I can help you with this issue, please find below some troubleshooting points than can solve this issue:

First of all Kindly note that:

• 1- Regarding multiple CAM calibration using Kalibr the results you get are exactly as the CAD model in the article: T= [[ 0.99999996 0.00007783 -0.00028918 -0.40306714] [-0.00007903 0.99999139 -0.00414943 0.01028023] [ 0.00028885 0.00414946 0.99999135 -0.00365584] [ 0. 0. 0. 1. ]] Baseline: 0.403214794416 [m]

• 2- The still CARLA IMU accelerometer and gyroscope noise parameters can be found in: sharepoint/Datasets/yaml_config_files/calibration/imu0.yaml and this data is calculated using Allan plot generation tool based on recording IMU messages for long hours.

In order to tackle the problem of finding the exact biases (bw, ba) of the CARLA simulated IMU:

• 1- Scientific One of the solutions is to use Kalibr camera-IMU that works on the cumulative B-Spline generative model with order n=6, I see in the results that when the number of the spline nodes is :

Initializing a pose spline with 15005 knots (100.000000 knots per second over 150.049997 seconds) [DEBUG] [1678870107.831322]: discrete time shift: -11 [DEBUG] [1678870107.833658]: cont. time shift: 0.0549999987706 [DEBUG] [1678870107.835281]: dT: 0.00499999988824 Time shift camera to imu (t_imu = t_cam + shift): 0.0549999987706 Initializing a pose spline with 15005 knots (100.000000 knots per second over 150.049997 seconds) Gravity was intialized to [-2.53208877 -2.38991653 9.16761956] [m/s^2] Orientation prior camera-imu found as: (T_i_c) [[ 0.97326135 0.03451553 0.22709255] [-0.06203114 0.99140744 0.11516693] [-0.2211662 -0.12617433 0.96703958]] Gyro bias prior found as: (b_gyro) [ 0.00516027 -0.02917348 0.01284601]

the initialization values are perfectly near to the values set in CARLA during the simulation, however, a problem happens when the spline is initialized with 15017 knots. This issue can be solved by trying to fine-tune the time-offset and the other parameters of the spline in command before running the camera-IMU calibration command.

• 2- Scientific Another solution is to use the AprilGrid calibration sequence with the BASALT calibration tool, which is also build on the cumulative B-spline generative model but in an efficient implementation. [https://github.com/VladyslavUsenko/basalt-mirror/blob/master/doc/Calibration.md]

• 3- Technical Another possible solution, you can use our multimodal data acquisition Python APIs and set the IMU intrinsic parameters and camera-IMU inter-sensor extrinsic parameters with the desired values.

I hope this in-detail illustration can provide you with the possible solution and I remain at your disposal for any further inquiries.

Best regards