Performance issues with IMU-camera calibration tool

[davidiglesias]

Hi all,

We have recently been trying to calibrate a stereo rig with 2 cameras + IMU using Kalibr. However, we have seen that, although the results are not horrible and are physically consistent (e.g. they approximate the translations between sensors to a couple of centimeters), the accuracy is quite low, both in terms of reprojection errors (with an average of around 0.8-1px and maximum values of around 5px) and accelerometer/gyroscope acceleration errors (accelerometer 5-15m^2/s^4, gyro 1-5rad^2/s^2). We have tried to rule out possible causes of inaccuracies with little success, so maybe this is something you have seen before and can provide some help.

Our system is composed of two Flir cameras and one IMU which are hardware-synchronized. Time does not seem to be obviously off (I would say the timestamps are consistent across devices to a few milliseconds, although we have not conducted any formal tests to verify this); the camera images present minimal motion blur, and corners are correctly extracted from all frames; the motion of the rig manages to excite all axes and is sufficiently long with no shocks, as per the sample dataset and other datasets provided by the Vision group at TU Munich; the camera intrinsics are obtained both using Kalibr and the Matlab calibration toolbox, and we have tried two models (pinhole, equidistant and radial-tangential); IMU measurements have no weird spikes.

Happy to provide additional info if something is missing.

[NikolausDemmel]

Some thoughts:

• to rule out time-offeset issues (I know you mentioned hardware sync, but better check to be sure), have you tried to also optimize for time? What offset does it estimate?
• How do you set the IMU noise sigmas? Have you tried with a range of different values?
• camera intrinsics/extrinsics calibration results look good and are consistent across different calibration runs?
• What kind of lenses do you have? If they are wide-angle, have you tried the omni model?

What would help to assess the situation is if you could share the calibration results for both camera and cam-imu calibration (i.e. the pdf report), and maybe also your IMU config and command line calls.

[davidiglesias]

Hi @NikolausDemmel,

First off, an update -- by fiddling with IMU noise values, we have been able to decrease accelerometer errors to 0.04m^2/s^4 and gyroscope errors to 0.1rad^2/s^2. Now, to answer your questions:

• we have tried optimizing for time. The results show an offset between cameras and IMU of around 4-9 milliseconds.

• we have played with different IMU noise values. The values from manufacturer do not show the best results. As I mentioned above, we have decreased acc/gyro errors by a decent amount with this. However, reprojection errors seem to have a lower bound that can not be crossed -- the lowest mean value has been 0.7px, and the lowest maximum has been around 3px. For reference, the IMU is an Xsens MTI-100.

• We have tried recalibrating cameras, and the results we have obtained present differences with previous runs in the third decimal place for the distortion coefficients, and differences in the tens for the intrinsics. The impact of using these different calibration files when running camera-IMU has been very marginal at best.

• We are not using a wide-angle lens.

Report and calibration .txt file attached. Thanks a lot for your help! results-imucam.txt report-imucam.pdf

[YilmazKadir]

Hi, I have the same issue. The results seem to be correct but reprojection error is always around 1.5 pixels. I am using ZED camera with Xsens MTi-3 IMU. I would be appreciated if you could help.

Report file: report-imucam-hoay.pdf

[floriantschopp]

Hi, The reprojection errors are indeed higher then what I would expect from such cameras. Some questions:

• How large are reprojection errors in camera intrinsics calibration? What camera resolutions are you using?
• We experienced a high dependency between calibration quality and calibration target. What target are you using?
• One more thing could be your synchronization strategy. You mention that cameras and IMU are hardware synchronized. Does this mean that they are triggered at the same time or also that the timestamps are captured accordingly (e.g. on the same microcontroller)?