Closed lenardxu closed 1 year ago
Does your IMU measure gravity / are you using the "raw" IMU readings? It looks like the bspline isn't able to recover the accelerations (see 3rd page of the report).
On Tue, Sep 13, 2022 at 4:32 AM lenardxu @.***> wrote:
Greetings! Currently I am taking advantage of your shared tool kalibr_calibrate_imu_camera to calibrate my visual(mono)-inertial(external)-sensor. However, after various tries and experiments, the results are still not realistic. The following link to the pdf is the best estimated result so far.
https://drive.google.com/file/d/1fPdSuu1gwYxdce8BECoUQiCGtcx54IpY/view?usp=sharing And the corresponding bag's link: https://drive.google.com/file/d/1OvPiSmlS2xILy9msvLfntwvBX0g6NFD8/view?usp=sharing For comparison, here is the link to the pdf whose result is worse.
https://drive.google.com/file/d/1sPRBIfmXFu6L8mLuUB7YIgLl9cF9E140/view?usp=sharing And the corresponding bag's link: https://drive.google.com/file/d/1LTiClE8NKyDpHmbXBfiq6tKcszYtYENH/view?usp=sharing
The corresponding setup is: I use OAK-D camera which doesn't support hardware/external trigger and only the left mono one is used, and BNO055 as an external Imu sensor. As a result, a software synchronization other than hardware synchronization is adopted for syncing both messages. The software sync in general performs linear interpolation upon the temporally nearest two Imu data at the instant when an image is captured, in order to acquire the approxiamted Imu data at that instant. The Imu sensor coordinate frames after validation is aligned with the camera's, i.e., z pointing to forward, x to right and y to downward. And the Imu's origin with respect to the camera's is roughly about (x, y, z) = (+3cm, +2.6cm, +1.5cm).
The Imu sensor is self-calibrated and further calibrated in terms of its noise parameters using allan_variance_ros tool recommended by you. And the camera sensor is re-calibrated using your multi camera calibration tool, which turns out to be good given the low reprojection error equal to around [-0.000000, 0.000000] +- [0.174857, 0.159922]pixel, or see the following:
Calibration results
Camera-system parameters: cam0 (/oakd_bno055_ros/left/image): type: <class 'aslam_cv.libaslam_cv_python.DistortedPinholeCameraGeometry'> distortion: [ 0.0750117 -0.17704149 0.00107357 -0.00171022] +- [0.01120927 0.02042603 0.00187764 0.00152155] projection: [429.79984321 430.77618923 324.10110114 198.35165358] +- [3.93954063 3.92397601 2.33407519 2.83935374] reprojection error: [-0.000000, 0.000000] +- [0.174857, 0.159922]
Target configuration
Type: aprilgrid Tags: Rows: 6 Cols: 6 Size: 0.024 [m] Spacing 0.0072 [m]
In my experiment corresponding to the result shown above, the white noise is doubled and bias walk multiplied by ten upon the result of allan_variance_ros in order to consider the temperature. As you see in the following link:
https://drive.google.com/file/d/1AdeAJZLSV5uch3bvQYOWFxo3MTUmmYDV/view?usp=sharing
The calibartion target in calibration of vi-sensor is the same as above in calibration of multiple-camera.
Could anyone give me any pointer how to improve the result? Many thanks in advance.
— Reply to this email directly, view it on GitHub https://github.com/ethz-asl/kalibr/issues/568, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQ6TYX6YE2WLPZYFSAEBU3V6A3ZTANCNFSM6AAAAAAQLGH7V4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>
As I use BNO055, which supports the so-called "NDOF" fusion mode, the IMU readings that are fed to kalibr tool are already filtered upon the built-in fusion algorithm. Hence, the acceleration data correspond to linear acceleration (i.e., the acceleration that is applied due to movement), the angular rate the same, and the orientation data correspond to the absolute orientation (i.e., relative to the earth and its magnetic field). Hence, do you mean that I need to feed the "raw" IMU readings in terms of acceleration (with gravity vector integrated) and angular rate data?
Your IMU needs to have gravity (this is what the IMU actually reads). Otherwise, the global inertial frame cannot be recovered.
On Tue, Sep 13, 2022 at 10:50 AM lenardxu @.***> wrote:
Does your IMU measure gravity / are you using the "raw" IMU readings? It looks like the bspline isn't able to recover the accelerations (see 3rd page of the report). … <#m6114731663067383078> On Tue, Sep 13, 2022 at 4:32 AM lenardxu @.> wrote: Greetings! Currently I am taking advantage of your shared tool kalibr_calibrate_imu_camera to calibrate my visual(mono)-inertial(external)-sensor. However, after various tries and experiments, the results are still not realistic. The following link to the pdf is the best estimated result so far. https://drive.google.com/file/d/1fPdSuu1gwYxdce8BECoUQiCGtcx54IpY/view?usp=sharing https://drive.google.com/file/d/1fPdSuu1gwYxdce8BECoUQiCGtcx54IpY/view?usp=sharing And the corresponding bag's link: https://drive.google.com/file/d/1OvPiSmlS2xILy9msvLfntwvBX0g6NFD8/view?usp=sharing https://drive.google.com/file/d/1OvPiSmlS2xILy9msvLfntwvBX0g6NFD8/view?usp=sharing For comparison, here is the link to the pdf whose result is worse. https://drive.google.com/file/d/1sPRBIfmXFu6L8mLuUB7YIgLl9cF9E140/view?usp=sharing https://drive.google.com/file/d/1sPRBIfmXFu6L8mLuUB7YIgLl9cF9E140/view?usp=sharing And the corresponding bag's link: https://drive.google.com/file/d/1LTiClE8NKyDpHmbXBfiq6tKcszYtYENH/view?usp=sharing https://drive.google.com/file/d/1LTiClE8NKyDpHmbXBfiq6tKcszYtYENH/view?usp=sharing The corresponding setup is: I use OAK-D camera which doesn't support hardware/external trigger and only the left mono one is used, and BNO055 as an external Imu sensor. As a result, a software synchronization other than hardware synchronization is adopted for syncing both messages. The software sync in general performs linear interpolation upon the temporally nearest two Imu data at the instant when an image is captured, in order to acquire the approxiamted Imu data at that instant. The Imu sensor coordinate frames after validation is aligned with the camera's, i.e., z pointing to forward, x to right and y to downward. And the Imu's origin with respect to the camera's is roughly about (x, y, z) = (+3cm, +2.6cm, +1.5cm). The Imu sensor is self-calibrated and further calibrated in terms of its noise parameters using allan_variance_ros tool recommended by you. And the camera sensor is re-calibrated using your multi camera calibration tool, which turns out to be good given the low reprojection error equal to around [-0.000000, 0.000000] +- [0.174857, 0.159922]pixel, or see the following: Calibration results ==================== Camera-system parameters: cam0 (/oakd_bno055_ros/left/image): type: <class 'aslam_cv.libaslam_cv_python.DistortedPinholeCameraGeometry'> distortion: [ 0.0750117 -0.17704149 0.00107357 -0.00171022] +- [0.01120927 0.02042603 0.00187764 0.00152155] projection: [429.79984321 430.77618923 324.10110114 198.35165358] +- [3.93954063 3.92397601 2.33407519 2.83935374] reprojection error: [-0.000000, 0.000000] +- [0.174857, 0.159922] Target configuration ==================== Type: aprilgrid Tags: Rows: 6 Cols: 6 Size: 0.024 [m] Spacing 0.0072 [m] In my experiment corresponding to the result shown above, the white noise is doubled and bias walk multiplied by ten upon the result of allan_variance_ros in order to consider the temperature. As you see in the following link: https://drive.google.com/file/d/1AdeAJZLSV5uch3bvQYOWFxo3MTUmmYDV/view?usp=sharing https://drive.google.com/file/d/1AdeAJZLSV5uch3bvQYOWFxo3MTUmmYDV/view?usp=sharing The calibartion target in calibration of vi-sensor is the same as above in calibration of multiple-camera. Could anyone give me any pointer how to improve the result? Many thanks in advance. — Reply to this email directly, view it on GitHub <#568 https://github.com/ethz-asl/kalibr/issues/568>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQ6TYX6YE2WLPZYFSAEBU3V6A3ZTANCNFSM6AAAAAAQLGH7V4 https://github.com/notifications/unsubscribe-auth/AAQ6TYX6YE2WLPZYFSAEBU3V6A3ZTANCNFSM6AAAAAAQLGH7V4 . You are receiving this because you are subscribed to this thread.Message ID: @.>
As I use BNO055, which supports the so-called "NDOF" fusion mode, the IMU readings that are fed to kalibr tool are already filtered upon the built-in fusion algorithm. Hence, the acceleration data correspond to linear acceleration (i.e., the acceleration that is applied due to movement), the angular rate the same, and the orientation data correspond to the absolute orientation (i.e., relative to the earth and its magnetic field). Hence, do you mean that I need to feed the "raw" IMU readings in terms of acceleration (with gravity vector integrated) and angular rate data?
— Reply to this email directly, view it on GitHub https://github.com/ethz-asl/kalibr/issues/568#issuecomment-1245527551, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQ6TYUHF6ZK7VFJZGMGSS3V6CIBXANCNFSM6AAAAAAQLGH7V4 . You are receiving this because you commented.Message ID: @.***>
Thanks! I've never thought about that. I'll then give it a shot to see any improvement.
As you proposed, I've selected the "raw" acceleration data instead of the linear acceleration data as input. The calibration accuracy is significantly improved, finally being realistic in terms of the transformation and gravity factor. The result is shown below. https://drive.google.com/file/d/1yavI03iWzwwRAqVrYT4ykXP2LWWEUfS4/view?usp=sharing But why is the gravity necessary to be included? Is there any resource introducing the related algorithm like paper that I can refer to? Sincerely thanks again for saving my day!
This is just part of the definition of the inertial model. I don't know if kalibr can be changed to support different gravities (such as different planets or in space).
This book provides some fundamental concepts if you wanted a research starting point:
Chatfield, Averil Burton. Fundamentals of high accuracy inertial navigation. Vol. 174. Aiaa, 1997.
Thanks for that recommendation! Unfortunately I have to continue registering another unsolved problem under this topic, since that problem is still around a unrealistic (or say inaccurate) estimation of translation parameters. First off, upon my manual measurement of distance between the geometirc center of Imu sensor (BNO055 Adafruit) and the optical center of mono camera (left camera of OAK-D), the distance along three axes in camera frame is roughly (x, y, z) ~ (0.036, 0.028, 0.014)m. As introduced at beginning of this issue, both frames have the same axes orientation, which follow the camera frame's convention. For that, here is a picture of how the visual-inertial sensor looks like in space. https://drive.google.com/file/d/1Q0UkkOs_tZx2f3ccXa9bWsKcW7ZIf8Xp/view?usp=sharing
Then, thanks to your proposal that I should take use of the "raw" acceleration data with gravity vector integrated, the estimated translation is much better than before when only linear acceleration is considered. However, that result still needs improvement compared to the "groundtruth" value shown above. Here are the following tries/experiments:
As the first try in improving the result, I tuned the imu noise density and bias random walk (x2,x10 or x4,x10, or x10,10) based on the estimated result of allan_variance_ros
tool. Among the results, I took the best one, which is shown in the link below.
https://drive.google.com/file/d/17WXrZ4Zk_o6s_rIRIgGsuFyfI-xbaqeF/view?usp=sharing
As you see, there is still a large deviation/drift in y-axis, about 4cm. The deviations in the other two axes are about 1cm, but still visible.
The corresponding rosbag file is here:
https://drive.google.com/file/d/1wqbw60CVkxU65UhB5811ubIrNULeZpJ4/view?usp=sharing
And for your convenience, the calibration video is extracted here:
https://drive.google.com/file/d/1mRm-dc_5uDMkEtjy2ZfapSucPPdkpV5e/view?usp=sharing
And the noise parameters of Imu (under 4 hours) is below:
https://drive.google.com/file/d/1NIJFIvrs-Vx5u7SAOX_Cd09pthhN4sNE/view?usp=sharing
Since there's no problem with camera calirbration using your multi-cam tool, I don't attach the result here again.
The command options I took use of is:
rosrun kalibr kalibr_calibrate_imu_camera --target calib_data/april_6x6_24x24mm.yaml --cam calib_data/oakd_left_400p_camchain.yaml --imu calib_data/imu.yaml --bag calib_data/calib_vi.bag --bag-from-to 0 68 --perform-synchronization --show-extract
Since among all the results after tuning noise sigma, the drift in y axis is equally large, I guess that the calibration algorithm may have a restriction on the spatial setup between Imu and camera. So upon the large deviation in y-axis, I move the Imu sensor along the negative y-direction so that the Imu is basically equal to mono camera in terms of y value. As a result, the distance between them is about (x, y, z) ~ (0.034, 0.001, 0.017)m. (It looks like this: https://drive.google.com/file/d/1IL3yTEk_WSRJ3RlADUNRV8Oy9PlnymL-/view?usp=sharing) However, the large drift in y-axis is still there, but with the moved distance removed. So my guess is wrong. But in the following experiments, I keep the spatially changed setup in this step for convenience since there should be no such restriction.
Then after retrieving some reference examples of VI sensor calibration, I found that the calibration procedure does not take a long time as mine (about 90s). So I accelerate that procedure a bit to see any improvement when the record is limited say under 60s. But still there is no improvement.
Finally, I found that the angular velocities error in three axes are not in the same "magnitude", as you may refer to the result in the first experiment: the error of angular velocities around x and y axis are smaller than the one around z axis. Hence, I increase the angular velocities around x and y axis. As a result, the result gets improved a bit, with the drift in y-axis reduced to 3.5cm, in x-axis to 0.4cm, and in z-axis to 0.1cm, as shown below: https://drive.google.com/file/d/1IrxLz9HoLcPp2eZH10Xby5naaEBPwDqh/view?usp=sharing But the drift in y-axis is still large. The corresponding bag file is below: https://drive.google.com/file/d/154ASOg4XvnHPVyWdT73kVFANPkzhL20P/view?usp=sharing And for your convenience the extracted video is below: https://drive.google.com/file/d/1qSl77C-84-_NVpUrLgxRhKotvmuLMFWi/view?usp=sharing
Therefore, I'd like to ask for your help @goldbattle, to see what is the problem here. Or if there's anyone offering any help, hearty thanks!
Try calibrating with the scale-misaligned IMU model, you might have a very bad IMU if you are seeing that the angular velocities are not correct. I would also make sure they are in radians per second and a right handed coordinate frame (these are ROS REP conventions).
First off, the unit of angular rate is double-checked after comparing to the corresponding register value documented in BNO055's datasheet, which turns out to be correct. And the IMU coordinate frame is also validated by observing the variation of linear acceleration and angular rate in specific motions in a reasonable way, which suggests that the frame follows right-hand rule.
Then by following your proposal, I tried scale-misaligned IMU model (option: scale_misalignment
) in calibration. The result is as follows:
https://drive.google.com/file/d/1PeZUK5xKnpQZ76B8rYYkCOPcbP-_LoED/view?usp=sharing
, which is based on the 4th experiment above that has achieved the best result. Not only do the angular velocities correspond to the motions in calibration procedure, but also surprisingly (at least for me), instead of observing a bad performance of gyro, the calibration result upon the scale-misaligned IMU model applied is better than the one upon calibrated (default) model, as you can see in the link above.
Hence, I guess that thoughts on improving accuracy of calibration before aren't valid any more. To validate on that and the possible improvement brought about by using scale-misaligned IMU model, I use the rosbag file in the 1st experiment above which has a smoother and less aggressive procedure. As a result, the estimated translation is much closer to the "groundtruth" one under the same experiment setup (x2,x10), with the deviation being roughly around (0.001, 0.003, 0.001)m. That could be the most realistic one. You may refer the result in the link below: https://drive.google.com/file/d/1aChpxa2QmBYBaJqNRzv0QwEsRNQe3HI2/view?usp=sharing
Upon the results above, I'd like to ask you three questions:
g
.
Greetings! Currently I am taking advantage of your shared tool
kalibr_calibrate_imu_camera
to calibrate my visual(mono)-inertial(external)-sensor. However, after various tries and experiments, the results are still not realistic. The following link to the pdf is the best estimated result so far. https://drive.google.com/file/d/1fPdSuu1gwYxdce8BECoUQiCGtcx54IpY/view?usp=sharing And the corresponding bag's link: https://drive.google.com/file/d/1OvPiSmlS2xILy9msvLfntwvBX0g6NFD8/view?usp=sharing For comparison, here is the link to the pdf whose result is worse. https://drive.google.com/file/d/1sPRBIfmXFu6L8mLuUB7YIgLl9cF9E140/view?usp=sharing And the corresponding bag's link: https://drive.google.com/file/d/1LTiClE8NKyDpHmbXBfiq6tKcszYtYENH/view?usp=sharingThe corresponding setup is: I use OAK-D camera which doesn't support hardware/external trigger and only the left mono one is used, and BNO055 as an external Imu sensor. As a result, a software synchronization other than hardware synchronization is adopted for syncing both messages. The software sync in general performs linear interpolation upon the temporally nearest two Imu data at the instant when an image is captured, in order to acquire the approxiamted Imu data at that instant. The Imu sensor coordinate frames after validation is aligned with the camera's, i.e., z pointing to forward, x to right and y to downward. And the Imu's origin with respect to the camera's is roughly about (x, y, z) = (+3cm, +2.6cm, +1.5cm).
The Imu sensor is self-calibrated and further calibrated in terms of its noise parameters using
allan_variance_ros
tool recommended by you. And the camera sensor is re-calibrated using your multi camera calibration tool, which turns out to be good given the low reprojection error equal to around [-0.000000, 0.000000] +- [0.174857, 0.159922]pixel, or see the following:In my experiment corresponding to the result shown above, the white noise is doubled and bias walk multiplied by ten upon the result of
allan_variance_ros
in order to consider the temperature. As you see in the following link: https://drive.google.com/file/d/1AdeAJZLSV5uch3bvQYOWFxo3MTUmmYDV/view?usp=sharingThe calibartion target in calibration of vi-sensor is the same as above in calibration of multiple-camera.
Could anyone give me any pointer how to improve the result? Many thanks in advance.