Add RGBD mode for OAK camera

[borongyuan]

I added RGBD mode for OAK cameras, but only those models with global shutter color cameras are supported. There are actually only 2 models in the supported sensors list, OV9782 and AR0234. Their resolutions happen to be consistent with the mono camera's resolution list, so adding list items can be avoided. I'm using OAK-D Pro W Dev, so I added the NG2094 board. As for AR0234, it is currently only used in OAK-D LR and needs to be tested and updated later.

Some adjustments have also been made to local transforms to make the configuration of different modes more consistent. Now the center camera is used as the camera base frame, and the left and right cameras and IMU will calculate offset based on this. This is also consistent with the URDF definition in depthai-ros. The data in URDF is actually not that official. It was actually what I measured last time.

I encountered some performance issues when three cameras are working simultaneously. This is a bit strange, because judging from depthai's debugging info, there is no problem with hardware resource allocation. But there are at least two ways to avoid it, one is to reduce the depth resolution, and the other is to disable subpixel. So when using a USB camera, the depth resolution is limited to 400P. In addition, v2.23.0 of depthai-core is required, otherwise there will be an issue with depth image alignment.

https://github.com/introlab/rtabmap/assets/22918715/2a78d8a9-5a26-4fe5-ade8-9cf7b95cb97e

[matlabbe]

Is the "depth image alignment" issue affecting all oak cameras?

We could add minimum 2.23 version when looking at depthai package here: https://github.com/introlab/rtabmap/blob/f56875db4ada45ce9e8564c6a3911ebd809b83bd/CMakeLists.txt#L647

Suggestion (removing the QUIET to see the cmake warning if older depthai version is detected):

FIND_PACKAGE(depthai 2.23)

CMake Warning at CMakeLists.txt:647 (FIND_PACKAGE):
  Could not find a configuration file for package "depthai" that is
  compatible with requested version "2.23".

  The following configuration files were considered but not accepted:

    /opt/ros/noetic/lib/x86_64-linux-gnu/cmake/depthai/depthaiConfig.cmake, version: 2.22.0

[borongyuan]

Is the "depth image alignment" issue affecting all oak cameras?

Specifically, it only affects depth-color alignment. Depth-mono alignment is not affected. We need the depth image to be perfectly aligned with undistorted color image. Generic camera node is used as @saching13 suggested.

Camera node will automatically undistort still, video and preview streams, while isp stream will be left as is.

Perfect depth-color alignment when using v2.23.0. When using v2.22.0, the alignment error at the edge of the images is obvious. Seems to be due to missing conversion of camera intrinsics.

We could add minimum 2.23 version when looking at depthai package here:

I think it's good to always follow the latest release of depthai-core. On the other hand, some users may directly use the precompiled version in the ROS repository to simplify installation. The version distributed by ROS is still at v2.22.0. Perhaps we can wait until it is updated.

They released v2.24.0 a few hours ago, which seems to be helpful in improving the data synchronization implementation.

[matlabbe]

It seems for Noetic and Iron, they are already at 2.23. However, it is still at 2.22 on Humble till next Humble release. Lets keep it like this then. Thanks for the PR!

[cdb0y511]

Hi,@borongyuan. I noticed that OAK series cameras have achieved excellent performance on real-time slam; I am new to these sensors, but if I want to buy one for 3D reconstruction/mapping, do you have any recommendations? OAK-D-Pro-W / OAK-D-Pro / OAK-S-2 /OAK-D-Lite?

[borongyuan]

Hi,@borongyuan. I noticed that OAK series cameras have achieved excellent performance on real-time slam; I am new to these sensors, but if I want to buy one for 3D reconstruction/mapping, do you have any recommendations? OAK-D-Pro-W / OAK-D-Pro / OAK-S-2 /OAK-D-Lite?

For VSLAM, it is recommended to use cameras with wide FOV. But this comes at the cost of reducing depth measurement accuracy, especially over long distances. If you want to use the depth image directly for 3D reconstruction, this may not be good. But if you want to fuse data from different camera poses, it's fine. In addition, they upgraded the production line late last month and used a new calibration method. This improves depth measurement accuracy, especially for wide FOV models. So it is recommended that you wait for a period of time, or consult them directly to ensure you get the product from the new batch.

Pro model has IR laser dot projector and IR illumination LED. But we haven’t been able to make good use of them yet. Illumination LED can provide better night vision capabilities, but for wide FOV models, it cannot cover the entire FOV. Dot projector provides active stereo vision, which can improve depth measurement accuracy, but also interferes with VSLAM feature extraction. I'm referring to this example to alternately switch dot projector on and off, but there are still some problems with the implementation of this part.

OAK-D-Lite does not have an IMU (although the doc says it does). I have confirmed with them that future batches will not have IMUs either. Therefore OAK-D-Lite can only be used for VO and not for VIO.

[cdb0y511]

Hi,@borongyuan. I noticed that OAK series cameras have achieved excellent performance on real-time slam; I am new to these sensors, but if I want to buy one for 3D reconstruction/mapping, do you have any recommendations? OAK-D-Pro-W / OAK-D-Pro / OAK-S-2 /OAK-D-Lite?

For VSLAM, it is recommended to use cameras with wide FOV. But this comes at the cost of reducing depth measurement accuracy, especially over long distances. If you want to use the depth image directly for 3D reconstruction, this may not be good. But if you want to fuse data from different camera poses, it's fine. In addition, they upgraded the production line late last month and used a new calibration method. This improves depth measurement accuracy, especially for wide FOV models. So it is recommended that you wait for a period of time, or consult them directly to ensure you get the product from the new batch.

Pro model has IR laser dot projector and IR illumination LED. But we haven’t been able to make good use of them yet. Illumination LED can provide better night vision capabilities, but for wide FOV models, it cannot cover the entire FOV. Dot projector provides active stereo vision, which can improve depth measurement accuracy, but also interferes with VSLAM feature extraction. I'm referring to this example to alternately switch dot projector on and off, but there are still some problems with the implementation of this part.

OAK-D-Lite does not have an IMU (although the doc says it does). I have confirmed with them that future batches will not have IMUs either. Therefore OAK-D-Lite can only be used for VO and not for VIO.

I appreciate your detailed explanation. I am now curious about this new batch.
What's the new mode name or code name (update for all OAK series or just special mode)? How to get it? I originally planned to purchase from Oak China Taobao, do they have this new batch? do you have any advice? Thanks again.

[borongyuan]

There are no new product models. It is just an upgrade to the factory calibration of existing products. The OAK-D W may be the main affected model, as its previous depth measurement accuracy was not very high.

I have also been paying attention to the impact of stereo calibration on VIO accuracy recently. Since the camera calibration is minimizing reprojection error, it is more sensitive to points at close range. The work in this paper is worthy of reference.

你可以直接咨询淘宝客服，如果他们不清楚的话可以让他们跟张总确认一下。

[Genozen]

@borongyuan Hi, I'm trying to follow your footstep in getting the Oak-D Pro W to work with stand-alone Rtabmap...

I have both DepthAI-Python and OpenVins built from source in there individual folders, but when I do cmake .. in rtabmap/build the result is as follow:

It doesn't seem like the DepthAI is found, nor is my OpenVins...

Any guidance regarding this?

[borongyuan]

You need to use depthai-core, which is a C++ library. If you build it from source, you need to set depthai_DIR so that RTAB-Map can find it. https://github.com/luxonis/depthai-core?tab=readme-ov-file#using-find_package If you also use ROS, an easier way is to install the precompiled ros-{rosdistro}-depthai from ROS repository. It can be found if you source the setup.bash of ROS distribution. For OpenVINS, you can also compile it as a ROS package. Then source the setup.bash of your workspace. Of course, ROS is not used when running stand-alone.

[Genozen]

@borongyuan , thank you for the swift reply. I managed to have CMake recognize the DepthAI and built it successfully. However, when selecting the Oak-D camera and new dataset, when I press start I end up with a crashed Rtabmap and terminal error:

Have you encountered something similar before?

[borongyuan]

For OAK-D S2 and OAK-D W, the available resolutions are 800P/720P/400P. The 480P is for OAK-D Lite.

[Genozen]

@borongyuan sorry to bother again

But my rtabmap keeps crashing...

Running it on: Jetson Orin Nano (Jetpack 6, Ubuntu 22.04) DepthAI ver. 2.23.0

When I click:

1. File -> New database -> Ignore -> Start (it crash)

When editing the resolution and starting again they all crash, even hitting the test button... (also only specific to selecting Oak-D type of camera)

There's a similar issue posted on Luxonis' repo, but my DepthAI works just fine independantly... https://github.com/luxonis/depthai-core/issues/499

[borongyuan]

I haven't tried JetPack 6 DP yet. I'm waiting for their production release. I did not encounter this issue on my laptop with Ubuntu 22.04 and OpenCV 4.5.4. Is your depthai-core built with OpenCV 4.8.0? If it's from the ROS repository, it should be linked to OpenCV 4.5.4. I generally avoid using OpenCV from Ubuntu repo on Jetson. I generally use OpenCV extracted from Nvidia's Docker images since it is built with CUDA support. But I haven't updated this part for JetPack 6 yet. Then build all ROS components that rely on OpenCV. https://github.com/dusty-nv/jetson-containers/blob/master/packages/opencv/config.py Because I saw that the memory trying to allocate in the error was too large. So this should be an OpenCV related issue.