How to evaluate the calibration result

[francescoli]

Hi, Nice work of this camera imu calibration tool. It's easy to use, thanks guys. I managed to get several calibration results, but I'm confused to choose one of them, since the transformation matrices are quite different. There is something wierd that the translation is too large for a mobile phone, like this T_ci: (imu to cam0): [m] [[-0.01867876 -0.99980259 -0.00677455 0.09131875] [-0.99980497 0.01872141 -0.00628717 0.28932638] [ 0.00641276 0.00665579 -0.99995729 -0.14433917] [ 0. 0. 0. 1. ]] By the way, the camera intrinsic calculated by kalibr_calibrate_camera is quite different from the result of the calibration toolbox for matlab under the link below, especially the distortion，does it matter? http://www.vision.caltech.edu/bouguetj/calib_doc/

[rehderj]

Hi, your results are indeed weird and suggest that there are some issues with your calibration. In general, cellphone IMUs pose a challenge since they are not "intrinsically" calibrated, i.e. they may exhibit considerable scale errors and misaligned axes. We will release an update that addresses these issues. Nevertheless, we achieved decent calibration performance for those devices as well using the toolbox in its current version.

So there are a couple of indicators to look at here:

• Please share your calibration result files with us. The residuals are often a good starting point to look for problems in the calibration.
• There are sometimes misdetections or lacking detections due to insufficient lighting or an inadequate size of the calibration pattern. The camera intrinsic calibration should be comparable to that given by Bouguet's toolbox. If this is not the case, there is something fishy and you should enable the options that visualize the target detections to check for errors there.
• Did you enable --time-calibration? This is not enabled by default, but fixed temporal offsets are a major source of errors in calibration. Please note that depending on your triggering scheme, the delay may be exposure dependent, so for best results, try to keep the exposure time of the camera constant. Again, the calibration results text file would contain information on whether this option was enabled.
• In general, mobile phones are rolling shutter cameras, and hence not perfectly explained by our global shutter model. While I would still expect decent results for sufficient lighting and not too rapid motions, please keep the problem in mind, it might haunt you at other stages of your processing pipeline.

I hope this helped. In brief, please check the calibration target detections and the time offset, they are the most likely source of error in failing calibrations. Please don't hesitate to post follow-up questions and please close this issue in case this solved the problem.

Cheers! Joern

[francescoli]

Hi, rehderj Thanks for your answer, first of all.

1. The calibration result shows the Gyro and Accelerometer error are too big

Calibration results
===================
Reprojection error squarred (cam0):  mean 0.334255672384, median 0.145422882332, std: 0.625160125909
Gyro error squarred (imu0):          mean 20.5386900081, median 0.958286163099, std: 234.186056878
Accelerometer error squarred (imu0): mean 215.669137081, median 71.8154754967, std: 310.528582206

Transformation (cam0):
-----------------------
T_ci:  (imu to cam0): [m]
[[-0.01867876 -0.99980259 -0.00677455  0.09131875]
 [-0.99980497  0.01872141 -0.00628717  0.28932638]
 [ 0.00641276  0.00665579 -0.99995729 -0.14433917]
 [ 0.          0.          0.          1.        ]]

T_ic:  (cam0 to imu): [m]
[[-0.01867876 -0.99980497  0.00641276  0.29190129]
 [-0.99980259  0.01872141  0.00665579  0.08684482]
 [-0.00677455 -0.00628717 -0.99995729 -0.14189532]
 [ 0.          0.          0.          1.        ]]

timeshift cam0 to imu0: [s] (t_imu = t_cam + shift)
0.0

Gravity vector in target coords: : [m/s^2]
[ 0.10272722 -9.80742544  0.19988348]

Calibration configuration
=========================

cam0
-----
  Camera model: pinhole
  Focal length: [506.1395188906223, 510.45625341047054]
  Principal point: [313.76367343475073, 233.37814335885324]
  Distortion model: equidistant
  Distortion coefficients: [0.5756276608306082, -1.1125234948898801, 3.110984752973998, -2.4596130156739475]
  Type: aprilgrid
  Tags: 
    Rows: 6
    Cols: 6
    Size: 0.0348 [m]
    Spacing 0.01044 [m]

IMU configuration
=================

  Update rate: 500.0
  Accelerometer:
    Noise density: 0.01 
    Noise density (discrete): 0.22360679775 
    Random walk: 0.0002
  Gyroscope:
    Noise density: 0.005
    Noise density (discrete): 0.111803398875 
    Random walk: 4e-06

1. I print the pattern on A3 paper and I modify the target yaml with the true size of tag
2. Haven't enable --time-calibration, I'll try in a minute

[francescoli]

with new data captured with slow motion and --time-calibration enabled, I got a new result

Calibration results
===================
Reprojection error squarred (cam0):  mean 0.505082514608, median 0.0776331018236, std: 3.7959497435
Gyro error squarred (imu0):          mean 5.84126495862, median 0.178412422449, std: 90.6932039277
Accelerometer error squarred (imu0): mean 594.835592177, median 201.568278555, std: 887.414898897

Transformation (cam0):
-----------------------
T_ci:  (imu to cam0): [m]
[[ 0.02177074 -0.99972688  0.00849741  0.00156212]
 [-0.9992679  -0.02202663 -0.03128091  0.00677226]
 [ 0.03145954 -0.00781018 -0.99947451 -0.0077898 ]
 [ 0.          0.          0.          1.        ]]

T_ic:  (cam0 to imu): [m]
[[ 0.02177074 -0.9992679   0.03145954  0.00697835]
 [-0.99972688 -0.02202663 -0.00781018  0.00165002]
 [ 0.00849741 -0.03128091 -0.99947451 -0.00758714]
 [ 0.          0.          0.          1.        ]]

timeshift cam0 to imu0: [s] (t_imu = t_cam + shift)
-0.0512026005661

Gravity vector in target coords: : [m/s^2]
[-0.13968464 -9.80521486  0.27267157]

Calibration configuration
=========================

cam0
-----
  Camera model: pinhole
  Focal length: [495.74503345983834, 499.2166115347462]
  Principal point: [321.3194140496275, 227.57614863303843]
  Distortion model: equidistant
  Distortion coefficients: [0.5558028769887173, -0.5767222943928554, -0.3056331553586644, 3.63584266762539]
  Type: aprilgrid
  Tags: 
    Rows: 6
    Cols: 6
    Size: 0.0348 [m]
    Spacing 0.01044 [m]

IMU configuration
=================

  Update rate: 200.0
  Accelerometer:
    Noise density: 0.01 
    Noise density (discrete): 0.141421356237 
    Random walk: 0.0002
  Gyroscope:
    Noise density: 0.01
    Noise density (discrete): 0.141421356237 
    Random walk: 4e-06

[francescoli]

By increasing the Noise density, I get a new result seems more reasonable. I've read the Wiki page about IMU Noise Model. However I can just find the Noise density of Gyroscope in the datasheet. So I leave Random walk and Noise density of Accelerometer as default. btw I don't know the exact serial number of the IMU, probably MPU-65XX by InvenSense

[rehderj]

Ok, thanks for sharing those results. There is still something clearly wrong with the accelerometer residuals.

A couple of different issues could cause that:

• I am not entirely sure how you obtained the noise characteristics of your IMU. Obviously, setting incorrect values may result in impaired calibrations. Please consider using an Allan Variance analysis to obtain more suitable values.
• Currently, the initial guess for the direction of gravity is hard coded (https://github.com/ethz-asl/kalibr/blob/master/aslam_offline_calibration/kalibr/python/kalibr_imu_camera_calibration/IccCalibrator.py#L34-L37). In rare cases, this value does not converge to a correct value for incorrectly aligned targets. In our update, we will include an initialization from data.
• Your IMU might return vastly incorrect values. Please plot the measurements to see whether there are obvious issues (spurious measurements, saturation, etc.) and double check the norm of the measurement at rest to determine obvious scale errors.

[rehderj]

I just saw your post. Using the method described in "Maximum Likelihood Identification of Inertial Sensor Noise Model Parameters", we identified the attached values (albeit for the MPU9150, so potentially with limited relevance to your application). Please consider using this as a starting point for further research into Allan Variance analysis and related approaches for identifying the parameters of the noise model.

[rehderj]

Note however, that these results should be considered as "lower bounds". It is common practice to inflate the values to somewhat account for unmodelled effects.

[francescoli]

report-imucam-nexus_6.pdf.zip

The plot of accelerometer seems incorrect. Is there any tool to do Allan deviation analysis?

[francescoli]

Hi, Joern You are right, I find plenty of incorrect accelerometer data in my csv file, caused by the asynchronization of sensor data aquisition, same as talked about in this topic https://groups.google.com/forum/#!topic/kalibr-users/dRoLkps0jWw After updating the raw data and modifying the initial guess of gravity to suite my use case, I get a much more reasonable result. Both txt and plot are in the attached zip file. Again, thanks for your help! imucam.zip

[rehderj]

Ok, I am glad you could resolve the issues. At first glance, the IMU looks oddly close to the camera. Can you reproduce the results on multiple datasets? In previous posts, you mentioned that you moved the phone slowly while recoding the dataset. While that mitigates the rolling shutter effect, it renders the displacement less well observable, since sufficient rotational excitation has to be present for estimating the offset. So please try to strike a balance between the two objectives. Maybe, researching a teardown of the device will also provide some insights on what displacements to expect. Anyway, please consider closing the issue in case you deem the problems solved. Cheers!

[francescoli]

Yes, I find they are too close, too. Thank you, Joern

[henrikf5]

Hi, I also have the problem that the displacement are too close on my samsung galaxy s6. Did you find a solution that gave your a more reasonable result?

[rehderj]

Apart from issues in the synchronization between camera and IMU and issues with the rolling shutter effect of the camera, this problem may also be related to IMU scale factor mismatches and axis misalignments.

We are about to release an update to kalibr that allows for calibrating IMU intrinsics. Please check back for updates and let us know whether it had an impact on your estimation.

[Narcissuscyn]

Hi!can you tell me how to get the imu.yaml?I can't find relative parameters in a datasheet!thanks!

[rehderj]

Please see https://github.com/ethz-asl/kalibr/wiki/IMU-Noise-Model#how-to-obtain-the-parameters-for-your-imu for further detail on that issue.

[liyinnb]

Hi just curious, is the IMU intrinsics calibration tool going to be released soon?

[fmohammadi2g]

Hi, I used Kalibr tool for camera calibration and by using different target I end up with different projection and distortion matrix. the reprojection error for asymmetric pattern is low but the uncertainty for projection matrix is high, on the other hand the reprojection error for aprilgrid is high but the uncertainty for projection matrix is low. I am confused how to evaluate the calibration result. Thanks.