Accurate Stereo-IMU calibration on ZED2 camera

yhabib29 commented 2 years ago

Hello, I am trying to calibrate the ZED2 camera using Kalibr, but I could not manage to get correct results for the stereo-IMU calibration ...

1. Stereo calibration:

I perform data acquisition using ZED SDK, the camera remains fixed while I move the target on all the image trying to variate rotation and scale. I also tuned the camera settings (exposure, saturation, hue, ...) to minimize motion blur and maximize the contrast on the target.

Recording parameters: HD 720p / 15 FPS / IMAGE_UNRECTIFIED
Calibration parameters: pinhole-radtan / April tag (80x80cm)

All the data are stored in a ROS bag without compression. The bag file is then reindexed using the command below: rosrun allan_variance_ros cookbag.py --input data.bag --output data_sorted.bag

I get the following results which seems to be correct: report-cam-data.pdf results-cam-data.txt (I launched it 3 times and I get the same results +/- 2 on the projection)

2. Stereo-IMU calibration:

On this sequence, the target remains fixed and I move the camera to excite all the axis (XYZ translations, roll, pitch, yaw, and random movements) while always keeping the target in sight. I acquire IMU data on a dedicated thread running as fast as possible to reach the sensor rate (400 Hz).

Recording parameters: HD 720p / 30 FPS / IMAGE_UNRECTIFIED
Sensor parameters: 400 Hz / CALIBRATEDDATA (Values are corrected from bias, scale and misalignment)_
Calibration parameters: pinhole-radtan / April tag (80x80cm) / Fine-tuned IMU parameters The IMU uses the image coordinate system: X-right, Y-down, Z-front

I first tried to get the IMU noise/bias parameters using allan_variance_ros toolkit. Here are the computed IMU config and stereo-IMU calibration results: imu_allan.yaml.txt results-imucam-data.txt report-imucam-data.pdf

Optimization problem initialized with 18649 design variables and 807019 error terms
The Jacobian matrix is 1684522 x 83902
[0.0]: J: 1.28187e+09
[1]: J: 2.86842e+07, dJ: 1.25318e+09, deltaX: 0.637571, LM - lambda:10 mu:2
...
[13]: J: 2.5401e+07, dJ: 0.00877449, deltaX: 0.336677, LM - lambda:0.180088 mu:2

As the last calibration did not fit the model, I fine-tuned the parameters to reach the following results: imu_custom_params4.yaml.txt results-imucam-data.txt report-imucam-data.pdf

Optimization problem initialized with 18649 design variables and 807019 error terms
The Jacobian matrix is 1684522 x 83902
[0.0]: J: 1.77579e+07
[1]: J: 2.92333e+06, dJ: 1.48346e+07, deltaX: 1.14925, LM - lambda:10 mu:2
...
[30]: J: 1.43827e+06, dJ: 121.186, deltaX: 0.00247999, LM - lambda:555.075 mu:2

As we can see in the PDF report, the reprojection error is really bad !

Questions:

Why are the reprojection error plots in the PDF report higher than the calibration results in the TXT file ? (for both Stereo and Stereo-IMU calibration)
How could I improve the Stereo-IMU calibration ?
Does the choice of the IMU coordinate system matters as long as it is right-handed ?

Thanks in advance for your help !

goldbattle commented 1 year ago

It looks like there is a huge spike at around 58 seconds into the dataset. It also looks like you are getting > 2G of accelerations in some places. I am not sure if this was realistics to your motion. The coordinate system needs to be right handed. This is a convention that is required.

yhabib29 commented 1 year ago

Thanks for your answer @goldbattle, and sorry I forgot to reply before. From your analysis, I think the problem must come from the ZED 2 IMU sensor ... I will further investigate in this direction.

ethz-asl / kalibr