High reprojection error during IMU calibration

[lsorgi]

Task: calibration of a system with 2 cameras synchronized + IMU

I noticed that there are cases where, despite the camera calibration completes with very low reprojection error, the following IMU/camera calibration starts with high reprojection error of the second camera.

ex.

Report after camera calibration

Calibration results 
====================
Camera-system parameters:
cam0 (/cam0/image_raw):
type: <class 'aslam_cv.libaslam_cv_python.EquidistantDistortedPinholeCameraGeometry'>
distortion: [ 0.13562678 -0.06020752  0.03139384 -0.01440789] +- [0.00587618 0.02394142 0.03726668 0.01953935]
projection: [330.0489308  329.98816425 303.07605656 199.87323138] +- [0.11827859 0.11692096 0.16477902
0.25446792]
reprojection error: [0.000000, -0.000001] +- [0.073164, 0.079186]

cam1 (/cam1/image_raw):
type: <class 'aslam_cv.libaslam_cv_python.EquidistantDistortedPinholeCameraGeometry'>
distortion: [-0.00327695  0.00206282 -0.00020782 -0.00001808] +- [0.00349016 0.00851597 0.00816563 0.0026737 ]
projection: [308.6984622  308.59007466 292.88845869 197.52478534] +- [0.14175268 0.14303774 0.1858457 
0.19185388]
reprojection error: [-0.000002, 0.000000] +- [0.067541, 0.080838

Log during IMU/Camera calibration

Before Optimization
===================
Normalized Residuals
----------------------------
Reprojection error (cam0):     mean 0.6949407117331186, median 0.6088699046243529, std: 0.4526829157360357
Reprojection error (cam1):     mean 7.972420158716779, median 0.8407079649614485, std: 21.246966735011373
Gyroscope error (imu0):        mean 64.06817999881288, median 51.75675644799013, std: 46.39086642481769
Accelerometer error (imu0):    mean 6.024933803383341, median 3.5847657085015263, std: 9.59647472404789

Residuals
----------------------------
Reprojection error (cam0) [px]:     mean 0.6949407117331186, median 0.6088699046243529, std: 0.4526829157360357
Reprojection error (cam1) [px]:     mean 7.972420158716779, median 0.8407079649614485, std: 21.246966735011373
Gyroscope error (imu0) [rad/s]:     mean 0.10414095768169129, median 0.08412909783127309, std: 0.07540700012476473
Accelerometer error (imu0) [m/s^2]: mean 0.4359699598774328, median 0.25939706776990284, std: 0.6944100693784437

Optimizing...
Using the block_cholesky linear system solver
Using the levenberg_marquardt trust region policy
Using the block_cholesky linear system solver
Using the levenberg_marquardt trust region policy

[lsorgi]

any idea why this happens?

[goldbattle]

Please provide the dataset that this happens on. What are the reprojections after optimization has been completed?