Suggestion for both hardware and software solution for a stable visual-inertial odometry in low textured and dynamic illumination enviroments

[BarzelS]

Hi, For some time now I have been trying to find a solution to the problem of visual-inertial odometry in low-textured and dynamic illumination environments, I have tried several methods in combination with the D435i but I see that in practice there is no good solution for this problem.

1. I understood that the T265 will give me a hardware based odometry but the fact that it has only rgb cameras and no infrared cameras with a projector will make it hard for it to perform odometry on low-textured and low illumination environments, isn't it ?
2. I've tried rtabmap - rgbd odometry with a lot of odom strategies but none gave me good enough results, I'm interested what is your suggestion on both hardware and software combination for a visual inertial odometry on a drone?
3. It seems that the IMU is very unstable, for example I've tried VINS-FUSION and on every small movement the imu would reflect in large drift? is it a cheap IMU in the D435i that is very sensitive to small movements? Thanks

[MartyG-RealSense]

1. I would recommend choosing only a 400 Series depth camera. These contain a Vision Processor D4 circuit board that can make real-time auto exposure adjustments based on current lighting conditions.

The newest RealSense D455 model has an RGB sensor that has a fast global shutter for the first time like the shutter on the depth sensing. Previous 400 Series models had a slower rolling shutter on the RGB sensor that made color images prone to blur during capture of fast motion.

https://github.com/IntelRealSense/librealsense/issues/6610

2. If the IMU is a concern for you, it may be worth pairing a 400 Series camera with the OpenVSLAM Visual SLAM software. It is not dependent on an IMU and it is extremely customisable in regard to the camera hardware that it can work with.

https://github.com/xdspacelab/openvslam

3. The IMU component in RealSense cameras does not have internal calibration, and so may not have zero acceleration values even when the camera is motionless at idle. This issue can be addressed on the D435i IMU by performing a software calibration on the IMU.

https://dev.intelrealsense.com/docs/imu-calibration-tool-for-intel-realsense-depth-camera

[BarzelS]

1. I would recommend choosing only a 400 Series depth camera. These contain a Vision Processor D4 circuit board that can make real-time auto exposure adjustments based on current lighting conditions.

The newest RealSense D455 model has an RGB sensor that has a fast global shutter for the first time like the shutter on the depth sensing. Previous 400 Series models had a slower rolling shutter on the RGB sensor that made color images prone to blur during capture of fast motion.

IntelRealSense/librealsense#6610

1. If the IMU is a concern for you, it may be worth pairing a 400 Series camera with the OpenVSLAM Visual SLAM software. It is not dependent on an IMU and it is extremely customisable in regard to the camera hardware that it can work with.

https://github.com/xdspacelab/openvslam

1. The IMU component in RealSense cameras does not have internal calibration, and so may not have zero acceleration values even when the camera is motionless at idle. This issue can be addressed on the D435i IMU by performing a software calibration on the IMU.

https://dev.intelrealsense.com/docs/imu-calibration-tool-for-intel-realsense-depth-camera

1. Thanks.
2. Actually I've tried OpenVSLAM as well but again the results were bad, a low textured scene can cause a big drift with it. That's exactly the places where IMU is needed. This is why I am looking for the right combination.
3. I did this calibration multiple times but still have zero acceleration values even when the camera is motionless at idle.

[MartyG-RealSense]

If you have the D435i's projector enabled, the semi-random dot pattern projected by it can aid the camera in analysing the depth information of surfaces with low texture or no texture, as the dots act as a texture source.

Have you tried Intel's tutorial for IMU equipped SLAM on D435i, please?

https://github.com/IntelRealSense/realsense-ros/wiki/SLAM-with-D435i

I note that you have tried VINS-FUSION. There is also VINS-RGBD that can be used with D435i.

https://github.com/STAR-Center/VINS-RGBD

A research paper has been published about its use with D435i.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6567327/

[BarzelS]

If you have the D435i's projector enabled, the semi-random dot pattern projected by it can aid the camera in analysing the depth information of surfaces with low texture or no texture, as the dots act as a texture source.

Have you tried Intel's tutorial for IMU equipped SLAM on D435i, please?

https://github.com/IntelRealSense/realsense-ros/wiki/SLAM-with-D435i

I note that you have tried VINS-FUSION. There is also VINS-RGBD that can be used with D435i.

https://github.com/STAR-Center/VINS-RGBD

A research paper has been published about its use with D435i.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6567327/

1. Yes I tried the opensource_tracking launch file and got large drift immediate in the beginning. I think something has changed with rtabmap or with realsense that it doesn't work anymore, have you tried it with the new realsense versions?

2. Yes I've tried VINS-RGBD and got the same bad results.

As I said it seems that there is a problem with my IMU usage, normal VO(no IMU) approaches seems to work fine until I'm getting to low textured scenes but when I combine the IMU(VIO approaches) I'm getting this drift, I'm suspecting that it something related to things that have changed with both realsense-ros and librealsense version regarding the IMU.

[MartyG-RealSense]

I'm not aware of changes made to how the D435i's IMU is handled. I recall that you have been having issues with your IMU in regard to calibration in another case that you and I have been going through.

https://github.com/IntelRealSense/realsense-ros/issues/1286

In regard to your question in that case about how you could not get yaw: a RealSense team member says that pitch and roll are supported in the rs-motion example. In the link below, a RealSense user programmed their own Python mechanism for yaw.

https://github.com/IntelRealSense/librealsense/issues/4391#issuecomment-510369366

[BarzelS]

I'm not aware of changes made to how the D435i's IMU is handled. I recall that you have been having issues with your IMU in regard to calibration in another case that you and I have been going through.

1286

In regard to your question in that case about how you could not get yaw: a RealSense team member says that pitch and roll are supported in the rs-motion example. In the link below, a RealSense user programmed their own Python mechanism for yaw.

IntelRealSense/librealsense#4391 (comment)

1. So do you say that I have a broken IMU? how can I ensure it?
2. Tried this example and got X(roll?) correct, Z(pitch?) correct and Y(yaw?) stuck on around ~180 no matter what movement I'm doing

[MartyG-RealSense]

Whilst it is extremely rare for RealSense hardware components to fail, it is not impossible. Going through both of your cases though (this one and #1286) I can't see anything that strongly suggests an IMU component failure. And the IMU is not likely to have a direct effect on whether a surface is recognised if it is low-textured or the illumination level is low.

You of course have the option of exploring pairing your depth camera with a T265. If you wish to discuss that with a RealSense support team member, please close this case and start a new one, as somebody else on the RealSense team handles T265 related cases.

[BarzelS]

Whilst it is extremely rare for RealSense hardware components to fail, it is not impossible. Going through both of your cases though (this one and #1286) I can't see anything that strongly suggests an IMU component failure. And the IMU is not likely to have a bearing on whether a surface is recognised if it is low-textured or the illumination level is low.

You of course have the option of exploring pairing your depth camera with a T265. If you wish to discuss that with a RealSense support team member, please close this case and start a new one, as somebody else on the RealSense team handles T265 related cases.

Its not that the IMU is related to wheter a surface is recognized if it is low-textured or the illumination level is low, its the fact that it gives a continuous evaluation of the movement in the short term in scenarios where the visual odometry gets lost

[MartyG-RealSense]

I see. I understand what you mean now. However, this is getting into topics that I am not able to provide support about. So you are more likely to get the answers that you need if you close this case and start a new case about IMU navigation specifically so that a member of the RealSense team who can handle such cases can assist you.

Please make sure to include mention of the T265 in the title or the opening question if you need T265 information, so that Intel know that the case will not be handled by me. I apologise that I could not be of more help in this particular instance.

[BarzelS]

I'm not aware of changes made to how the D435i's IMU is handled. I recall that you have been having issues with your IMU in regard to calibration in another case that you and I have been going through.

1286

In regard to your question in that case about how you could not get yaw: a RealSense team member says that pitch and roll are supported in the rs-motion example. In the link below, a RealSense user programmed their own Python mechanism for yaw.

IntelRealSense/librealsense#4391 (comment)

I think I found something that has changed along the realsense versions, I played some old bag from the netwrok that was recorded with the D435i(https://star-center.shanghaitech.edu.cn/seafile/d/0ea45d1878914077ade5/ - Handheld Normal.bag) and then "rostopic echo /camera/imu"

1. The frame_id published for this topic was : "camera_imu_frame"
2. The new frame_id in my camera: "camera_imu_optical_frame"

Why you did this change? is there any way I can publish the imu in the camera_imu_frame and not in the new frame?

[MartyG-RealSense]

Your question above is something that I do not have knowledge of (I am on the Support team, not the development team). @doronhi Could you assist @shirbarzel with the question above please? Thanks!

[MartyG-RealSense]

@shirbarzel Whilst answering another RealSense user's SLAM question, I recalled a D435i compatible SLAM software by MIT SPARK Lab called Kimera and found the link to the details.

https://support.intelrealsense.com/hc/en-us/community/posts/360043774354-Highlighting-a-tutorial-for-Metric-Semantic-SLAM-with-Kimera-and-RealSense-D435i-

At 11 minutes 25 seconds into the YouTube video at the above link, it says that Kimera can be installed without ROS, though it takes longer (30 minutes) and more skill than an easy ROS based installation (5 minutes to begin compilation, 20 minutes to complete).

[BarzelS]

@shirbarzel Whilst answering another RealSense user's SLAM question, I recalled a D435i compatible SLAM software by MIT SPARK Lab called Kimera and found the link to the details.

https://support.intelrealsense.com/hc/en-us/community/posts/360043774354-Highlighting-a-tutorial-for-Metric-Semantic-SLAM-with-Kimera-and-RealSense-D435i-

At 11 minutes 25 seconds into the YouTube video at the above link, it says that Kimera can be installed without ROS, though it takes longer (30 minutes) and more skill than an easy ROS based installation (5 minutes to begin compilation, 20 minutes to complete).

As you can see here: https://github.com/MIT-SPARK/Kimera-VIO-ROS/issues/67 I've tried this package as well and had a lot of problems but thanks.

[BarzelS]

Your question above is something that I do not have knowledge of (I am on the Support team, not the development team). @doronhi Could you assist @shirbarzel with the question above please? Thanks!

Hi @MartyG-RealSense @doronhi, any updates?

[MartyG-RealSense]

@shirbarzel There is nothing new that I know of to add on that question, unfortunately.

[MartyG-RealSense]

I will close this case now, as there is unfortunately no further information that can be provided.