kalibr kalibr_calibrate_imu_camera Optimization failed!

[ericdonatyy]

Hi,

I am running imu calibration code with gopro camera. I am encountering this error. Although I searched for the reason for this, I could not find the cause of the error. Is the projection error too high? I'm waiting for your help.

Command: rosrun kalibr kalibr_calibrate_imu_camera --bag /home/bilal/Desktop/son5.bag --cam /home/bilal/Desktop/33/camchain.yaml --imu /home/bilal/Desktop/33/sensor.yaml --target chess.yaml

Output:

Estimating imu-camera rotation prior

Initializing a pose spline with 4990 knots (100.000000 knots per second over 49.900000 seconds)
Gravity was intialized to [ 6.10753059 -7.18059688  2.70287282] [m/s^2]
  Orientation prior camera-imu found as: (T_i_c)
[[ 0.50455949  0.45908115  0.73120737]
 [ 0.33669752  0.67523424 -0.65627243]
 [-0.79501855  0.57732419  0.18612436]]
  Gyro bias prior found as: (b_gyro)
[-0.0004817   0.00231137 -0.00081416]

Initializing a pose spline with 10004 knots (200.000000 knots per second over 50.020000 seconds)

Initializing the bias splines with 10004 knots

Adding camera error terms (/cam0/image_raw)
  Added 361 camera error terms                               

Adding camera error terms (/cam1/image_raw)
  Added 360 camera error terms                               

Adding accelerometer error terms (/imu0)
  Added 7220 of 7220 accelerometer error terms (skipped 0 out-of-bounds measurements)

Adding gyroscope error terms (/imu0)
  Added 7220 of 7220 gyroscope error terms (skipped 0 out-of-bounds measurements)

Before Optimization
===================
Normalized Residuals
----------------------------
Reprojection error (cam0):     mean 0.650465331697, median 0.529347114831, std: 0.507975482041
Reprojection error (cam1):     mean 27.4362298354, median 25.9955081109, std: 6.28544820982
Gyroscope error (imu0):        mean 6.56829362183, median 2.71905509518, std: 9.28832138299
Accelerometer error (imu0):    mean 83.1351784489, median 72.0552187392, std: 40.2044209746

Residuals
----------------------------
Reprojection error (cam0) [px]:     mean 0.650465331697, median 0.529347114831, std: 0.507975482041
Reprojection error (cam1) [px]:     mean 27.4362298354, median 25.9955081109, std: 6.28544820982
Gyroscope error (imu0) [rad/s]:     mean 0.0229890276764, median 0.00951669283312, std: 0.0325091248405
Accelerometer error (imu0) [m/s^2]: mean 1.2969087838, median 1.12406141233, std: 0.627188967204

Optimizing...
Using the block_cholesky linear system solver
Using the levenberg_marquardt trust region policy
Using the block_cholesky linear system solver
Using the levenberg_marquardt trust region policy
Initializing
Optimization problem initialized with 30032 design variables and 51078 error terms
The Jacobian matrix is 116594 x 120118
[0.0]: J: 5.38219e+14
Exception in thread block: [aslam::Exception] /home/bilal/kalibr_workspace/src/kalibr/aslam_nonparametric_estimation/aslam_splines/src/BSplineExpressions.cpp:447: toTransformationMatrixImplementation() assert(_bufferTmin <= _time.toScalar() < _bufferTmax) failed [1.60968e+09 <= 1.60968e+09 < 1.60968e+09]: Spline Coefficient Buffer Exceeded. Set larger buffer margins!Exception in thread block: 
[aslam::Exception] /home/bilal/kalibr_workspace/src/kalibr/aslam_nonparametric_estimation/aslam_splines/src/BSplineExpressions.cpp:447: toTransformationMatrixImplementation() assert(_bufferTmin <= _time.toScalar() < _bufferTmax) failed [1.60968e+09 <= 1.60968e+09 < 1.60968e+09]: Spline Coefficient Buffer Exceeded. Set larger buffer margins!
Exception in thread block: [aslam::Exception] /home/bilal/kalibr_workspace/src/kalibr/aslam_nonparametric_estimation/aslam_splines/src/BSplineExpressions.cpp:447: toTransformationMatrixImplementation() assert(_bufferTmin <= _time.toScalar() < _bufferTmax) failed [1.60968e+09 <= 1.60968e+09 < 1.60968e+09]: Spline Coefficient Buffer Exceeded. Set larger buffer margins!
[ERROR] [1672050004.889677]: std::exception
[ERROR] [1672050004.892747]: Optimization failed!
Traceback (most recent call last):
  File "/home/bilal/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_calibrate_imu_camera", line 247, in <module>
    main()
  File "/home/bilal/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_calibrate_imu_camera", line 209, in main
    iCal.optimize(maxIterations=parsed.max_iter, recoverCov=parsed.recover_cov)
  File "/home/bilal/kalibr_workspace/src/kalibr/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccCalibrator.py", line 180, in optimize
    raise RuntimeError("Optimization failed!")
RuntimeError: Optimization failed!

[goldbattle]

It looks like the cam1 has super large reprojection errors (bad static calibration?). Are you able to run with a single camera?

[ericdonatyy]

Dear @goldbattle, yes, I also run with cam0, and cam0 is 0.65 px error in stereo, cam0 becomes 2.051730257 px error when run mono-imu . Translation slightly increased. But it still hasn't offered real value. I need your help on this.

`Calibration results

Normalized Residuals

Reprojection error (cam0): mean 2.051730257, median 1.68190511365, std: 1.56539884023 Gyroscope error (imu0): mean 4.49433304664, median 3.84980693429, std: 2.92043330078 Accelerometer error (imu0): mean 16.0373827815, median 15.1064904411, std: 7.90494316191

Residuals

Reprojection error (cam0) [px]: mean 2.051730257, median 1.68190511365, std: 1.56539884023 Gyroscope error (imu0) [rad/s]: mean 0.000359546643731, median 0.000307984554743, std: 0.000233634664062 Accelerometer error (imu0) [m/s^2]: mean 0.268145040107, median 0.252580520175, std: 0.132170649667

Transformation (cam0):

T_ci: (imu0 to cam0): [[ 0.64357822 0.59596732 0.48023955 -0.19011274] [ 0.48790852 0.16398185 -0.85735362 -0.00905322] [-0.58970531 0.78608709 -0.18524238 0.08497719] [ 0. 0. 0. 1. ]]

T_ic: (cam0 to imu0): [[ 0.64357822 0.48790852 -0.58970531 0.17688106] [ 0.59596732 0.16398185 0.78608709 0.04798608] [ 0.48023955 -0.85735362 -0.18524238 0.09927922] [ 0. 0. 0. 1. ]]

timeshift cam0 to imu0: [s] (t_imu = t_cam + shift) -0.00888701085624

Gravity vector in target coords: [m/s^2] [ 9.63055457 1.08724449 -1.49624228]

Calibration configuration

cam0

Camera model: pinhole Focal length: [1425.9845132347257, 1426.118370418552] Principal point: [1238.7500720328392, 905.1767901828307] Distortion model: equidistant Distortion coefficients: [0.2857372589036288, -0.2281659693698818, 0.414673022500735, 0.08497884215003768] Type: checkerboard Rows Count: 6 Distance: 0.1 [m] Cols Count: 9 Distance: 0.1 [m]

IMU configuration

IMU0:

Model: scale-misalignment Update rate: 400.0 Accelerometer: Noise density: 0.000836 Noise density (discrete): 0.01672 Random walk: 6.9e-05 Gyroscope: Noise density: 4e-06 Noise density (discrete): 8e-05 Random walk: 1e-07 T_ib (imu0 to imu0) [[ 1. 0. 0. 0.] [ 0. 1. 0. 0.] [ 0. 0. 1. 0.] [ 0. 0. 0. 1.]] time offset with respect to IMU0: 0.0 [s] Gyroscope: M: [[ 0.00015373 0. 0. ] [-0.01060834 0.01633681 0. ] [-0.28007989 0.57167143 0.15081014]] A [(rad/s)/(m/s^2)]: [[ 0.0000065 -0.00001197 0.00002552] [ 0.00145075 0.00155677 -0.00497331] [-0.00021693 -0.0009993 0.00201586]] C_gyro_i: [[ 0.93197653 0.34683431 0.10547847] [-0.3623575 0.89993196 0.24252733] [-0.01080665 -0.2642507 0.96439348]] Accelerometer: M: [[ 0.22803719 0. 0. ] [-0.10098693 0.0772605 0. ] [-0.09600555 -0.54834283 0.85024363]]`

[goldbattle]

Likely your IMU is bad, or noises are incorrect here. Post a bag and the commands you are running and ensure you are running the master branch.

[XinZhanhua]

Hello, have you ever solved the problem? I have the same problem.

[XinZhanhua]

Likely your IMU is bad, or noises are incorrect here. Post a bag and the commands you are running and ensure you are running the master branch.

do you mean it's because the calibration of IMU is bad?

[XinZhanhua]

@ericdonatyy Can you please tell me how do you solved the first problem? I got the same error of the first one. I'm using a camera of 60HZ and IMU of 200HZ.