Breaking Changes

• 1/7/2024:
  • Removed javascript requirement. Please install py_gpmf_parser from now on.
  • Fixed Dockerfile and updated Readme with new installation instructions
  • Updated dependencies to Ceres 2.1.0
  • Updated pyTheiaSfM version --> Install new version (either master or 69c3d37).

OpenICC: An Open IMU and Camera Calibrator

I developed this repository to experiment with the accurate calibration of action cameras (e.g. GoPro cameras) to use them for geometric vision tasks like Structure-from-Motion, Photogrammetry and SLAM. Modern action cameras are equipped with various sensors like IMUs (accelerometer, gyroscope and magnetometer) and GPS. However the calibration data (e.g. camera projection and IMU to camera transformations) is not available.

This is where the OpenImuCameraCalibrator comes in. With this toolbox you can:

• Calibrate the intrinsics of your a GoPro camera. Supported camera models are:
  • Fisheye [6]
  • Division Undistortion [5]
  • Field-of-View [3]
  • Double Sphere [2]
  • Extended Unified [4]
  • Pinhole
  • Pinhole with radial-tangential distortion
• Extract the meta data integrated in the MP4 video file (called telemetry data)
• Calibrate the Camera to IMU rotation matrix and find the dataset dependent time offset
• Perform full continuous time batch optimization to find the full transformation matrix between IMU and camera
• Do an intrinsic calibration of your IMU using the method described in [11]
• [Experimental] Calibrate the rolling shutter line delay (not really working yet)

Results

This section provides some results for my two GoPro cameras (6 and 9). You can use this to verify your own results or use them as initial values for your application. So far I have been setting them to FullHD with wide FoV and 30/60 fps. This is probably the most common setting that people use.

Camera Calibration

IMU to Camera Calibration

Some SLAM Examples using ORB-SLAM3

GoPro9_25fps_1080

GoPro9_50fps_1080

GoPro9_25fps_1440_maxlens_fisheye

ORB-SLAM3 fork

Installation instructions

Tested on Ubuntu 18.04 and 20.04. and 22.04

1. Clone and build OpenCV >= 4.5.0 with contrib modules. Latter are needed for Aruco marker detection. On Ubuntu 22.04 you can also just install it from apt:

   sudo apt-get install libopencv-dev libopencv-contrib-dev

2. Install ceres 2.1

   git clone https://github.com/ceres-solver/ceres-solver
   git checkout 2.1.0
   mkdir -p build && cd build && cmake .. -DBUILD_EXAMPLES=OFF -DCMAKE_BUILD_TYPE=Release
   sudo make -j install

3. Clone and build the TheiaSfM fork.

   git clone https://github.com/urbste/pyTheiaSfM
   cd pyTheiaSfM && git checkout 69c3d37 && mkdir -p build && cd build
   cmake .. && make -j
   sudo make install

4. Build this project

   git clone https://github.com/urbste/OpenImuCameraCalibrator
   mkdir -p build && cd build && cmake ..
   make -j

5. Create a python >3.5 environment (or use your local python - not recommended)

   pip install -r requirements.txt

Dockerfile

Build the docker container

docker build -t openicc .

Now you can mount the OpenICC folder to your docker container, as well as the folder that contains your calibration data (e.g. download GoPro9 dataset to /home/Downloads/GoPro9 )

docker run -it --rm -v `pwd`:/home -v /home/Downloads/GoPro9:/dataset openicc

Finally you can run the calibration like this:

cd /home
python3 python/run_gopro_calibration.py --path_calib_dataset /dataset/dataset3/ --path_to_build ../OpenImuCameraCalibrator/build/applications/

Usage examples

• Calibrate a GoPro9
• Calibrate a SamsungS20FE
• Calibrate a GoPro IMU intrinsics
• Estimate a GoPro IMU noise parameters

Acknowlegements

This library would not have been possible without these great OpenSource projects:

• TheiaSfM - Camera models and optimization
• [Basalt-Headers]() - Spline implementation and optimization
• Lie Group Cumulative B-Splines - Lie Splines
• InertialScale - IMU to camera time offset and rotation matrix initialization
• OpenCV - Well OpenCV ;)
• Sophus - C++ Lie groups
• GoPro-Telemetry - Great JavaScript GoPro telemetry extractor
• Kontiki - Spline error weighting, VISFM
• IMU-TK - Static multi pose imu calibration
• Allan variance - Gyro noise characterization

  Literature and Code

  Libraries

  • [1] Theia Multiview Geometry Library: Tutorial & Reference

    Camera models:

  • [2] The Double Sphere Camera Model, V. Usenko and N. Demmel and D. Cremers, In 2018 International Conference on 3D Vision (3DV)
  • [3] FoV model: F. Devernay and O. Faugeras. Straight lines have to be straight. Machine vision and applications, 13(1):14–24, 2001.
  • [4] Extended unified: B. Khomutenko, G. Garcia, and P. Martinet. An enhanced unified camera model. IEEE Robotics and Automation Let- ters, 1(1):137–144, Jan 2016.
  • [5] Division undistortion: "Simultaneous linear estimation of multiple view distortion" by Andrew Fitzgibbon, CVPR 2001.
  • [6] Fisheye: Kannala, Juho, and Sami S. Brandt. "A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses." IEEE transactions on pattern analysis and machine intelligence 28.8 (2006): 1335-1340.

IMU calibration:

• [11] D. Tedaldi, A. Pretto and E. Menegatti, "A Robust and Easy to Implement Method for IMU Calibration without External Equipments". In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2014), May 31 - June 7, 2014 Hong Kong, China, Page(s): 3042 - 3049

Misc:

• [7] Mustaniemi J., Kannala J., Särkkä S., Matas J., Heikkilä J. "Inertial-Based Scale Estimation for Structure from Motion on Mobile Devices", International Conference on Intelligent Robots and Systems (IROS), 2017
• [8] Efficient Derivative Computation for Cumulative B-Splines on Lie Groups, C. Sommer, V. Usenko, D. Schubert, N. Demmel, D. Cremers, In 2020 Conference on Computer Vision and Pattern Recognition (CVPR)
• [9] Larsson, Viktor, Zuzana Kukelova, and Yinqiang Zheng. "Making minimal solvers for absolute pose estimation compact and robust." Proceedings of the IEEE International Conference on Computer Vision. 2017.
• [10] Hannes Ovrén and Per-Erik Forssén Spline Error Weighting for Robust Visual-Inertial Fusion In Proceedings of the IEEE on Computer Vision and Pattern Recognition (CVPR) June 2018

Working on / ToDo / Contributions Welcome

v0.1

• [x] Code cleanup, add license header
• [x] use different initialization for linear mode
• [x] Use more generic JSON meta data interface, so that others cameras can be calibrated that come with different telemetry formats
• [x] Write example for the calibration of a Smartphone
• [x] Rolling shutter calibration -> first resolve problems with templated spline functions
• [x] Add readout time as a optimizable parameter -> not working well, probably residual weighting needs to be improved
• [x] Allan variance -> imu_utils
• [x] Support AprilTag board from Kalibr to record same datasets
• [x] Calibrate axis misalignment and so on with MutliPoseOptimization from imu_tk
• [x] Include imu axis and scale in spline error functions
• [x] Beautify logs
• [x] Cleanup

v0.2

• [x] Model bias over time with a R3 spline
• [ ] Pose estimation with UPNP or MLPnP to support arbitraty camera types. Right now: undistortion and then using vanilla PnP --> will lead to problems for Ultra Wide Angle fisheye lenses (e.g. GoPro Max or potentially Max lens mod)
• [ ] Pose estimation with RSPnP
• [ ] use only bearings in spline reproj error -> local tangent reprojection error
• [ ] Accurate checkerboard detector (OpenCV has findCheckerboardSB. I integrated a first version but results were weird. Re-check...)
• [ ] Extend to multi-camera systems
• [x] Docker?
• [ ] Add more camera models -> Scaramuzza omni model, ...
• [x] Integrate updated version of spline optimizer

misc

• [ ] Put together a little paper on how this all works

Citation

If this tool helped you and you are using it in your work consider citing it as follows for now:

@misc{OpenICC,
  author = {Steffen Urban},
  title = {OpenICC: An Open IMU and Camera Calibrator},
  howpublished = "\url{https://github.com/urbste/OpenImuCameraCalibrator}",
}