D405 Image delay

[r91andersson]

I'm experiencing huge delays when using align_depth.enable:=true on our new D405 camera. Prior to the test I updated the firmware on the camera and installed ROS2 Foxy together with ROS wrapper and Realsense SDK. The versions I'm using: Platform: Nvidia Xavier NX Jetson 5.0.1, Ubuntu 20.04 ROS Foxy RealSense ROS v4.51.1 LibRealSense v2.51.1

I've seen the exact same problem with another camera (D435) but on ROS1 on the same Jetson Platform: https://github.com/IntelRealSense/librealsense/issues/8003

The issue is simple to reproduce, just launch an Realsense Depth Camera (D405,D435,D435i etc.):

ros2 launch realsense2_camera rs_launch.py align_depth.enable:=true

Then look at the delay on the color stream: ros2 topic delay /camera/color/image_rect_raw

And you'll get values at approx. 300ms

The most interesting thing is that if you put your hand in front the depth module and cover it, you'll see that the delay rapidly goes down to normal 30-40ms. And as soon as you remove your hand, the delay starts to increase again back to > 300ms.

I will try repeat the same steps but on a different platform (amd64).

[MartyG-RealSense]

Hi @r91andersson Are you still using the same movable camera module from https://github.com/IntelRealSense/realsense-ros/issues/1581 that you had large delay with in 2020 due to Global Time being enabled, please? If you are, does setting global_time_enabled to false again like in that previous case correct the delay?

Having said that, the symptoms that you describe are characteristic of a problem with exposure, as covering the camera with the hand during an image problem when auto-exposure is enabled can enable the auto-exposure to correct the problem. However, if the cause of the exposure problem is still present (such as the camera being faced directly towards the sun) then the problem can re-occur after hand removal.

[r91andersson]

Nope, currently I'm just holding the camera in my hand while moving it. Unfortunately, disable global_time_enabled makes no difference.

I tried disable the auto-exposure then subsequently setting the exposure level to 1 (min), but the delay is still around 170 ms on the topic: /camera/aligned_depth_to_color/image_raw

I'm currently running the depth and rgb with the profile 640x480x30.

Can you confirm the same issue exists on your side, by taking random Intel Realsense Depth camera with the align_depth:=true @MartyG-RealSense?

[MartyG-RealSense]

My computer is not equipped to run tests with ROS, unfortunately. I do apologize.

When you built the librealsense SDK on Jetson, did you install it from Jetson packages using the instructions at the installation_jetson.md page at the link below?

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md

Or did you build librealsense from source code with CMake?

When building librealsense with packages on Jetson, CUDA support is automatically enabled. But if librealsense is built from source on Jetson then the build flag -DBUILD_WITH_CUDA=TRUE should be included in the CMake build instruction so that librealsense's CUDA support is enabled. This enables depth-color alignment to be accelerated on Jetson by offloading processing from the Jetson's CPU to its Nvidia graphics GPU.

[r91andersson]

When I run on my PC (amd64) with the following spec:

  Ubuntu 22.04 
  ROS Iron
  RealSense ROS v4.54.1
  LibRealSense v2.54.1

I get normal delays on the image stream of approx. 40 ms (auto exposure on) when using align_depth.enable := true

When run on a different PC (amd64) with the following spec:

  Ubuntu 20.04 
  ROS Foxy
  RealSense ROS v4.51.1
  LibRealSense v2.51.1

I get delays of approx 20 ms (auto exposure on) when using align_depth.enable := true

[r91andersson]

My computer is not equipped to run tests with ROS, unfortunately. I do apologize.

When you built the librealsense SDK on Jetson, did you install it from Jetson packages using the instructions at the installation_jetson.md page at the link below?

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md

Or did you build librealsense from source code with CMake?

When building librealsense with packages on Jetson, CUDA support is automatically enabled. But if librealsense is built from source on Jetson then the build flag -DFORCE_RSUSB_BACKEND=TRUE should be included in the CMake build instruction so that librealsense's CUDA support is enabled. This enables depth-color alignment to be accelerated on Jetson by offloading processing from the Jetson's CPU to its Nvidia graphics GPU.

I used the apt-get install commands to install Realsense SDK: sudo apt install ros-foxy-librealsense2*

So I will try remove that and install it according to the link you provided. I'll get back with the result! Thanks

[MartyG-RealSense]

You are very welcome. I look forward to your next update. Good luck!

[r91andersson]

I did a reinstall according to the Jetson guide:

1. sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE || sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE
2. sudo add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo $(lsb_release -cs) main" -u
3. sudo apt-get install librealsense2-utils
4. sudo apt-get install librealsense2-dev

Then I installed the ROS2-wrapper: sudo apt install ros-foxy-realsense2-*

But still, when using align_depth.enable:=true, I get delays of 160 ms.

[r91andersson]

I just saw that OpenCV is not built with CUDA, could this be a problem? The output is from jtop

[r91andersson]

I tried build Librealsense SDK from source with Native Backend and build with CUDA enabled. RealSense ROS v4.51.1 LibRealSense v2.51.1 However, I'm still seeing large delays on the camera stream (approx 200-300ms).

It seems that when running the realsense node, the GPU isn't utilized att all according to JTOP, a bit suspect? Shouldn't the GPU be active while running the align_depth.enable:=true?

[r91andersson]

I know that on the previous librealsense versions, this was not a problem, it became a problem after the release of the SDK version v.2.50.0. Prior to that there was an acceptable delay.

I have another NX card with this setup: Ubuntu 18.04 RealSense ROS v2.3.1 LibRealSense v2.50.0 where the delay is OK. For that version, I installed the SDK via sudo apt-get install as this:

1. sudo apt-key adv --keyserver keyserver.ubuntu.com  --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE || sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key
2. sudo add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo $(lsb_release -cs) main" -u
3. sudo apt-get install librealsense2-utils librealsense2-dev -y

[r91andersson]

Is it possible to add the server key for the v.2.50.0 release and install the SDK together with the ROS-wrapper v4.51.1? @MartyG-RealSense

[MartyG-RealSense]

It would be possible to install librealsense 2.50.0 and the ROS wrapper together from packages at the same time in ROS1 but I cannot see a way to do so in ROS2 unfortunately.

It should be possible to build librealsense 2.50.0 and wrapper 4.51.1 separately from source code though. There is a 'forked' version of the 4.51.1 wrapper created by a RealSense ROS user at the link below that could be built insead of using the official wrapper repository.

https://github.com/HedgeHao/realsense-ros

[r91andersson]

Thanks @MartyG-RealSense.

I built Librealsense v.2.50.0 from source. Then I downloaded ROS wrapper v.4.0.4 (which is the one that is compatible with v.2.50.0) and built that from source. However, when I ran the rs_launch.py, the ROS wrapper did not publish the algined_depth images. I looked at the release notes and found out that align_depth was fixed in the upcoming release v.4.51.1, but that release doesn't work with Librealsense v.2.50.0.

So what I did was:

1. Build the v.4.0.4 
2. Manually merged the PR that made the algin_depth work in the upcoming release: https://github.com/IntelRealSense/realsense-ros/pull/2332
3. Rebuild workspace
4. Now align_depth works and we have an image delay of approx. 27 ms.

[MartyG-RealSense]

It's great to hear that you made significant progress through your creative solution!

[r91andersson]

Moreover, I found out that building Librealsense from source, then installing ROS-wrapper with sudo apt-get install...will overwrite the installed Librealsense version compiled from source.

[MartyG-RealSense]

Hi @r91andersson Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.