Large translation error in Extrinisic calibration between D415 cmera image and IMU

[ahar]

Hello,

We have tried to use the Kalibr package to perform extrinsic calibration of a D415 realsense camera and an IMU. A sample dataset is provided with this link. It also contains the calibration result we obtained. Upon carefully looking at the results we realize that the translation components contained within T_ci reflects a large distance between the IMU and the camera, which does not match our hardware setup. The error is around ten times the original numbers. Based on these observations and our experience using this package the following are the queries I have and prompt response on the same will help me a lot to understand this software better. Queries:

1. Are the coordinate frames of the camera and the IMU required to be the same? As ROS assumes right-handed coordinate system, I assume that in whatever orientation I keep these two devices, the data collected will be in individual body frames maintaining the right-handed system. Is that right?
2. If my assumptions in (1) are correct, how are the coordinate frames adjusted? I mean either the camera's or the IMU's frame must be considered as the reference right?
3. While going through some of the existing issues I observed that the delay between the camera and the IMU data plays a role and it is suggested to maintain the delay below 20 msec, which is for the data uploaded in the link above. What could be other reasons behind the wrong transformation?

Your help is much appreciated.

p.s. You may need to open your web browser in a private window (incognito) to view the shared files in the above link.

Best,

[ahar]

@goldbattle could you please help resolve my queries?

[goldbattle]

Sorry, have very minimal throughput this time of the year. Here are some initial comments.

1. Yes, everything should be right handed. If you have a left handed IMU publisher then the driver will need to be fixed. Personally, I have calibrated the T265 and D455 with no issues with Intel's driver.
2. The result in the yaml is for a specific direction, but the results.txt file has both transforms (in IMU or in CAMERA) in plan english.
3. If your secondary system can handle a time offset, then it is fine to have this offset. But 20ms is very large and you might want to check that your timestamps are good and monotonic.

I would point you in another direction from briefly looking at your results. You need to calibrate your IMU noises and inflate them a bit (2x for white, and 10x for walks).

• https://github.com/ethz-asl/kalibr/wiki/IMU-Noise-Model#how-to-obtain-the-parameters-for-your-imu
• https://github.com/ori-drs/allan_variance_ros

For example these should be "in bounds"

And your reprojection error should be much much lower than 5 pixel. I wouldn't expect any good performance if it is this large....

[ahar]

Hello @goldbattle Based on your initial response we looked into time-offset from the recorded bag file shared in my first post. I found that the camera data was reporting at 15Hz instead of at 30 Hz. In that, timeoffset exceeded 20ms (approx. 35Hz). Upon recording the dataset once more while ensuring the camera frame rate of 30Hz (reference earlier results), the calibration steps were performed again.

The results obtained for Xsens-630 IMU + D415 can be found here (unscaled noise). The same for Lord-CV7 IMU+ D415 can be found here. Both the IMUs were intrinsically calibrated beforehand using Allan-Variance-Ros package. Imu.yaml files for individual IMU containing random-walk and noise-density can be found in the abovementioned link respectively.

As you suggested, the extrinsic IMU+camera calibration with scaled-up noise parameters are also performed and can be found in link1 (Xsens-630) and link2 (LORD CV7).

_For Set-1 (Xsens630Imu with D415):_ In the calibration results with scaled-up noise parameters, time-offset appears to be 8.25ms. Also, the translation components in imu0 to cam0 Extrinsic transformation matrix has now come down to sub-cm level except for the first element of the translation vector. We expect it to be around 1cm whereas the calibration result shows approx. 5.2 cm. ***I would like to know your observations on it (screenshot attached).

With unscaled noise parameters link of the IMUs the optimization failed by printing that matrix is not positive definite. (screenshot below) :      

Queries:

1. Why did the optimization fail in this case?

_For Set-2 (Lord-CV7 Imu with D415):_ In this case, the calibration process was completed for both the Scaled and Unscaled Imu.yaml file. But the first element of the translation vector appears incorrect in this case too link Here, the time shift is around 5.4ms, and it will be good to know your views as to what is the source of the error in this case.

[ahar]

@goldbattle any views on my above query?

[goldbattle]

The failure is likely due to the noises being too small causing numerical issues, but I am not sure.