search

IntelRealSense / librealsense

Intel® RealSense™ SDK
https://www.intelrealsense.com/
Apache License 2.0
7.53k stars 4.81k forks source link

FPS drop and stream latency on Jetson ROS #9879

Open nkorenicFR opened 2 years ago

nkorenicFR commented 2 years ago
Required Info
Camera Model D435i
Firmware Version 5.12.07.100
Operating System & Version L4T-32.4.4
Kernel Version (Linux Only) 4.9.140-tegra
Platform Jetson Xavier AGX
SDK Version 2.49.0
Language ROS melodic + realsense ROS wrapper 2.3.1
Segment Robot

Issue Description

Stream FPS drop and latencies (hundreds of microseconds up to a few seconds) occur in ROS Jetson when pointcloud or align depth is enabled. The issue can be reproduced by running roslaunch realsense2_camera rs_camera.launch. Below are given some cases which reproduce the issue. FPS drop was measured with rostopic hz <topic> and latencies are observed by displaying stream with rosurun rviz rviz and rosrun image_view image_view image:=<image topic>.

enable pointcloud align depth filters RGB Hz depth Hz Comment
false false - 30 30 -
true false - 10 24 issue present
true false colorizer 9 22 issue present
true false decimation 30 30 -
true false disparity 17 30 issue present
true true - 20 40 issue present
true true colorizer 16 27 issue present
true true decimation 12 34 issue present
true true disparity 11 45 issue present
false true - 30 30 -
false true colorizer 24 36 issue present
false true decimation 30 30 -
false true disparity 17 40 issue present

LibRealSense

Realsense is built from source with CUDA support cmake .. -DBUILD_EXAMPLES=true -DCMAKE_BUILD_TYPE=release -DFORCE_RSUSB_BACKEND=false -DBUILD_WITH_CUDA=true. Native kernel drivers are used and the L4T kernel is patched with ./scripts/patch-realsense-ubuntu-L4T.sh.

ROS launch

Lunch enables two camera streams: RGB (1280x720, RGB8, 30FPS) and, depth (848x480, Z16, 30FPS) The launch is built and run with librealsense v2.49.0.

CPU load

On the issue, the system has increased CPU consumption (sometimes one core is fully loaded). Our Jetson AGX system is configured to full power mode (30W) with all cores enabled.

CPU load when pointcloud and align depth used ros_big_cpu_load

CPU load when pointcloud and align depth not used ros_normal_CPU_load

Reproducing in RealSense Viewer

In the viewer, similar could be reproduced when GLSL processing and rendering were disabled. Is it possible that ROS uses LibRealSense without GLSL support?

Similar issues

A similar issue has already been reported #9519, but it is closed without an adequate solution.

MartyG-RealSense commented 2 years ago

Hi @nkorenicFR The RealSense ROS wrapper version used should ideally be matched as closely as possible with the librealsense SDK version listed for that particular wrapper in its release notes. The librealsense version to match with wrapper 2.31 is 2.48.0. There is not a wrapper for matching with 2.49.0, and the next wrapper version should be for upcoming librealsense version 2.50.0.

In regard to GLSL, it is not enabled in RealSense applications by default and has to be deliberately incorporated into programs, like it is in the RealSense Viewer.

nkorenicFR commented 2 years ago

Hi @MartyG-RealSense ,

Thank you for your fast response. I will try with librealsense 2.48.0. Regarding GLSL in ROS wrapper, i guess that realsense2-gl module (https://github.com/IntelRealSense/librealsense/pull/3654) is not used, as i could not locate rs2::gl::pointcloud or rs2::gl::colorize. Could we expect better ROS performances (FPS, stream latency) by adding GLSL to the RealSense ROS wrapper?

MartyG-RealSense commented 2 years ago

I am not personally aware of anyone who has tried adjusting the rs2:: instructions in the ROS wrapper to convert them to rs2::gl:: instructions. The key lines in the wrapper's base_realsense_node.cpp to look at for each type of rs2:: operation are likely to be the ones below (I say 'likely' as I am not involved in the wrapper's development).

Pointcloud https://github.com/IntelRealSense/realsense-ros/blob/development/realsense2_camera/src/base_realsense_node.cpp#L1273

Align https://github.com/IntelRealSense/realsense-ros/blob/development/realsense2_camera/src/base_realsense_node.cpp#L1253

Colorizer https://github.com/IntelRealSense/realsense-ros/blob/development/realsense2_camera/src/base_realsense_node.cpp#L1258

nkorenicFR commented 2 years ago

Same behavior with RealSense SDK 2.48.0, the issue is still present. Any idea what else could we try?

Does Intel have any benchmark results on ROS jetson that can be used for comparison? Is this expected behavior on the Jetson platform?

MartyG-RealSense commented 2 years ago

Hi @nkorenicFR Dips in performance when enabling pointcloud or align in the RealSense ROS wrapper with Jetson boards is an issue that has previously been reported by a small number of Jetson users, as mentioned in https://github.com/IntelRealSense/librealsense/issues/9519#issuecomment-888942705

There are not ROS benchmark results available from Intel that compare pointcloud and align performance on Jetson with non-Jetson systems. Intel did publish a CPU performance table of alignment with CUDA versus non-CUDA when CUDA support was first introduced in the librealsense SDK though. That table can be found at the link below.

https://github.com/IntelRealSense/librealsense/pull/2670