Incorrect camera-imu calibration on Android phone

[deephanson94]

Hi, I am using Kalibr to get the imu-to-camera rotation matrix of my Android phone, following the steps in Kalibr wiki pages. Both the cam0 and imu0 data are collected by running an Android app using openCV and Android's SensorEvent API on the phone.

The T_ic i obtained from running kalibr gives me such results:

However according to Android's motion sensor axis, both the camera and IMU should have the following axis:

Looking at the z-axis alone, clearly the pic above shows that the transformation should have a 180 degrees rotation along the x-axis, instead of 90 degrees as calculated by Kalibr. We verify the T_ic by running a SLAM system, showing that using Kalibr's calculated results gives us completely off results, while using the calculated R_ic from the 2nd picture gives us a much better one.

Any reason why Kalibr is giving an incorrect result? Thanks!

[floriantschopp]

Could you provide us with the calibration report pdf?

[deephanson94]

Hi, below is the calibration report. Thanks!

Calibration results
===================
Reprojection error squarred (cam0):  mean 3.18261607125, median 0.247172719517, std: 11.4006035858
Gyro error squarred (imu0):          mean 43.8825374008, median 2.18418476787, std: 160.968098486
Accelerometer error squarred (imu0): mean 3.873398121, median 1.72076076904, std: 4.57913447349

Transformation (cam0):
-----------------------
T_ci:  (imu to cam0): [m]
[[ 0.98992233 -0.11850583  0.07752521  0.18354507]
 [ 0.11253415  0.3259813  -0.93865449 -0.11998504]
 [ 0.08596426  0.93791927  0.33603211 -0.41874077]
 [ 0.          0.          0.          1.        ]]

T_ic:  (cam0 to imu): [m]
[[ 0.98992233  0.11253415  0.08596426 -0.13219621]
 [-0.11850583  0.3259813   0.93791927  0.45360908]
 [ 0.07752521 -0.93865449  0.33603211  0.01385648]
 [ 0.          0.          0.          1.        ]]

timeshift cam0 to imu0: [s] (t_imu = t_cam + shift)
0.0

Gravity vector in target coords: : [m/s^2]
[ 3.0382659  -1.38191559  9.22471407]

Calibration configuration
=========================

cam0
-----
  Camera model: pinhole
  Focal length: [477.66323403861907, 476.7905844796118]
  Principal point: [321.5718465928593, 243.80161697418322]
  Distortion model: radtan
  Distortion coefficients: [0.018925400673184004, 0.005034125915214071, 0.001434022944766616, 0.0012693538389572246]
  Type: aprilgrid
  Tags:
    Rows: 6
    Cols: 6
    Size: 0.025 [m]
    Spacing 0.007 [m]

IMU configuration
=================

  Update rate: 400.0
  Accelerometer:
    Noise density: 0.01
    Noise density (discrete): 0.2
    Random walk: 0.02
  Gyroscope:
    Noise density: 0.005
    Noise density (discrete): 0.1
    Random walk: 0.0004

[floriantschopp]

Could you also link the report pdf produced by Kalibr? It's way easier to investigate problems there.

Best, Flo

[deephanson94]

Update: I think it failed to generate the pdf due to an import failure ImportError: No module named apport.fileutils Seems like a CDE problem https://github.com/ethz-asl/kalibr/issues/315. However i dont have sudo permission to sudo apt-get install..

Hi Flo,

I dont see any pdf output in my dir. Could you suggest on how to generate the pdf? I'm quite new to Kalibr :)

Thanks!

[floriantschopp]

The CDE distribution has some bugs related to generating a full report I head, never used it myself. I would suggest you to either build from source or you could also send me your bag (and corresponding parameters you used) and I can quickly run it on my system.

[deephanson94]

Sure thing. This is how i run it:

kalibr_calibrate_imu_camera --bag kalibr_dataset_2.bag --cam camchain_dataset_2.yaml --imu imu.yaml --target aprilgrid.yaml

The .bag/.yaml files can be extracted from the zip file: https://drive.google.com/file/d/1opyMJ2_RV91xte3IPkwZrxdPALz5pzYq/view?usp=drivesdk

[floriantschopp]

OK, I run the bag through the pipeline (also with time estimation which is rather often a problem). The results are here: camera calibration and Cam-IMU calibration. While the camera calibration looks reasonable (although not the best coverage of observations), the cam-imu calibration is very bad (also huge resulting reprojection error). Looking at the IMU and estimated bodyspline plot, it looks like the data is not really corresponding to each other. From the bag file, I saw that images are only available for a part of the whole measurement duration (see rqt_bag).

I see different possible problems:

• During the recording, the timestamps of images and IMU measurements are messed up resulting in non-corresponding measurements. This is also visible if you playback the bag in both rqt_plot and image_view e.g.:
  Probably a good way to check that is to look at some data live (e.g. with rqt_plot) and verify the data comes in with minimal delay.
• The IMU could also be wrongly scaled. This could also result from a wrongly scaled April grid (during camera calibration, a wrong scale could be compensated with the focal length)! Here again, verify a reasonable scale with simple tests in rqt_plot with live data.

Hope this helps. Let me know if I can assist further.

[deephanson94]

Thanks @floriantschopp for the insights! It is very helpful indeed :) I will take a look on my IMU measurements and see if there's any issue with it.

I think the problem comes from the data collection from my Android phone. Do you have any suggestion on tools or code for img/IMU data collection from an Android phone for calibration purpose? I wrote it myself so I'm not sure if it's entirely correct.

[floriantschopp]

No, I have no experience with Android but visual-inertial synchronization with limited access to hardware can be very challenging. In terms of suggestions, you can have a look at VersaVIS, maybe you can reuse some of it for your code.

[deephanson94]

Alright, thanks for your kind suggestions!

[GabHoo]

https://github.com/OSUPCVLab/mobile-ar-sensor-logger

I know I am late, but in case anyboy still needs a good tool to record imu and camera datas from smartphone here it is!