search

IntelRealSense / realsense-ros

ROS Wrapper for Intel(R) RealSense(TM) Cameras
http://wiki.ros.org/RealSense
Apache License 2.0
2.59k stars 1.76k forks source link

Empty pointcloud data when accelerate_gpu_with_glsl = True #3206

Open edgarcamilocamacho opened 1 month ago

edgarcamilocamacho commented 1 month ago

Hello

Required Info
Camera Model D435i
Firmware Version 5.16.0.1
Operating System Ubuntu 20.04.6 LTS
Platform NVIDIA Jetson Orin AGX
Jetpack 5.1.1 [L4T 35.3.1]
Kernel Version (Linux Only) 5.10.104-tegra
Librealsense SDK Version 2.56.1
ROS Distro Humble
RealSense ROS Wrapper Version 4.56.1

Issue Description

When I set accelerate_gpu_with_glsl = True (at the begining of the execution, not dinamically), I get empty data pointcloud like:

header:
  stamp:
    sec: 1726699016
    nanosec: 654941162
  frame_id: camera_depth_optical_frame
height: 1
width: 0
fields:
- name: x
  offset: 0
  datatype: 7
  count: 1
- name: y
  offset: 4
  datatype: 7
  count: 1
- name: z
  offset: 8
  datatype: 7
  count: 1
is_bigendian: false
point_step: 16
row_step: 0
data: []
is_dense: true

But with accelerate_gpu_with_glsl = False it works perfectly.

Maybe I'm setting something wrong, I attach the list of ros params:

rs-params.txt

MartyG-RealSense commented 1 month ago

Hi @edgarcamilocamacho In rare cases on some computers, a pointcloud will disappear when GLSL is enabled. There is no way to predict in advance which computers will be unable to use GLSL, unfortunately.

If you are using a Jetson then the usual way to accelerate a pointcloud will be to enable the librealsense SDK's CUDA support. This can be done by installing the Jetson version of the librealsense packages on the installation_jetson.md instruction page or by building from source code with the flag -DBUILD_WITH_CUDA=ON included in the CMake build instruction.

edgarcamilocamacho commented 1 month ago

So if I install librealsense and realsense-ros in the Jetson, both from apt packages, like:

I should get both of them running with GPU, right?

There is a way to identify that it's actually using the GPU?

edgarcamilocamacho commented 1 month ago

Following that idea I this in my Jetson container (the sumarized steps from the links I sent before):

# apt-key adv --keyserver keyserver.ubuntu.com --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE || sudo apt-key # adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE
# add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo $(lsb_release -cs) main" -u
# apt-get install -y librealsense2-utils librealsense2-dev ros-humble-realsense2-*

After that I get these versions installed:

# apt list --installed *realsense*
Listing... Done
librealsense2-dev/focal,now 2.55.1-0~realsense.3336 arm64 [installed]
librealsense2-gl/focal,now 2.55.1-0~realsense.3336 arm64 [installed,automatic]
librealsense2-udev-rules/focal,now 2.55.1-0~realsense.3336 arm64 [installed,automatic]
librealsense2-utils/focal,now 2.55.1-0~realsense.3336 arm64 [installed]
librealsense2/focal,now 2.55.1-0~realsense.3336 arm64 [installed,automatic]
ros-humble-librealsense2/focal,now 2.54.1-1focal.20231218.194516 arm64 [installed,automatic]
ros-humble-realsense2-camera-dbgsym/focal,now 4.54.1-1focal.20231230.021252 arm64 [installed]
ros-humble-realsense2-camera-msgs-dbgsym/focal,now 4.54.1-1focal.20231229.233803 arm64 [installed]
ros-humble-realsense2-camera-msgs/focal,now 4.54.1-1focal.20231229.233803 arm64 [installed]
ros-humble-realsense2-camera/focal,now 4.54.1-1focal.20231230.021252 arm64 [installed]
ros-humble-realsense2-description/focal,now 4.54.1-1focal.20231229.234552 arm64 [installed]

I got librealsense2=2.55.1 and realsense-ros=4.54.1, so aparently the apt packages are pointing to not compatible libraries.

I could install X.54.1 in both cases but then I won't have the accelerate_gpu_with_glsl (or that's what I understand, because that parameters is only mentioned in the documentation from version 4.55.1).

edgarcamilocamacho commented 1 month ago

Wait... seeing it closely, it installed librealsense2 and ros-humble-librealsense2. The second one has the correct version, so the questions would be:

MartyG-RealSense commented 1 month ago

I am not involved in the compilation and publishing of packages, so do not have advice to provide about their contents.

As a general rule, assume that if installation is not being performed with the Jetson packages on theinstallation_jetson.md page or the JetsonHacks website's installLibrealsense.sh build script at the link below then the packages do not contain CUDA support. This is because packages that will be used on desktop / laptop PCs do not need the CUDA support because it only works with Nvidia GPUs, usually the ones built onto a Jetson board.

https://github.com/JetsonHacksNano/installLibrealsense

MartyG-RealSense commented 1 month ago

Hi @edgarcamilocamacho Do you have an update about this case that you can provide, please Thanks!

edgarcamilocamacho commented 1 month ago

These are the steps I'm following in my dockerfile (in the Jetson). I took the steps from the scripts you told me.

# Install librealsense
RUN cd /tmp && \
    wget https://github.com/IntelRealSense/librealsense/archive/refs/tags/v2.56.1.tar.gz && \
    tar -xzf v2.56.1.tar.gz && \
    cd librealsense-2.56.1 && \
    mkdir build && \
    cd build && \
    cmake .. \
        -DBUILD_PYTHON_BINDINGS:bool=true \
        -DPYTHON_EXECUTABLE=/usr/bin/python3 \
        -DBUILD_EXAMPLES=true \
        -DCMAKE_BUILD_TYPE=release \
        -DFORCE_LIBUVC=true \
        -DFORCE_RSUSB_BACKEND=false \
        -DBUILD_WITH_CUDA:bool=true \
        && \
    make -j 10 && \
    make install && \
    cd ../.. && \
    rm -r v2.56.1.tar.gz librealsense-2.56.1/

# Install realsense-ros
RUN source /setup_env.bash && \
    mkdir -p /robot/additional_ros_packages/src && \
    cd /robot/additional_ros_packages/src && \
    git clone -b 4.56.1  https://github.com/IntelRealSense/realsense-ros.git && \
    cd /robot/additional_ros_packages && \
    colcon build --merge-install --cmake-args '-DBUILD_ACCELERATE_GPU_WITH_GLSL=ON' && \
    rm -rf src build log

And I still have the same problem.

I attach the complete dockerfile.

In addition this is the docker-compose configuration:

  cameras:

    # common
    network_mode: host
    pid: host
    privileged: true
    runtime: nvidia
    restart: unless-stopped
    # specific
    image: gary.core.cameras.cameras:jetsonorin.4.3.beta

    environment:
      # display
      - XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR}
      - DISPLAY=:0

    env_file:
    # common
      - ./config/env/env

    volumes:
      # common
      - ./config:/robot/config
      - ./generic_persistent_data:/robot/generic_persistent_data
      - ./component_persistent_data/cameras:/robot/component_persistent_data
      # devices
      - /dev:/dev
      - /run/udev:/run/udev:ro
      # display
      - ${XDG_RUNTIME_DIR}/gdm/Xauthority:/root/.Xauthority
      - /tmp/.X11-unix:/tmp/.X11-unix:rw

rs.dockerfile.txt

MartyG-RealSense commented 1 month ago

My knowledge of Docker is admittedly limited, so there is not much that I can add to the previous advice, unfortunately.

I would recommend removing the -DBUILD_ACCELERATE_GPU_WITH_GLSL=ON pointcloud acceleration parameter, since the pointcloud will already be being accelerated by -DBUILD_WITH_CUDA:bool=true when this Dockerfile is used on Jetson.

Havibg said that, there have been a small number of past cases where problems disappear when not setting -DBUILD_WITH_CUDA to true. So you could try it either with -DBUILD_ACCELERATE_GPU_WITH_GLSL=ON or with DBUILD_WITH_CUDA:bool=true but not have both set to true at the same time.

MartyG-RealSense commented 1 month ago

Hi @edgarcamilocamacho Do you require further assistance with this case, please? Thanks!

edgarcamilocamacho commented 1 month ago

Hello @MartyG-RealSense , thanks for the fast response and sorry for the slow testing, we have several things on the table, but for sure next week we'll build the container again and test.

MartyG-RealSense commented 1 month ago

It's no problem at all. I look forward to your next report. Good luck!

MartyG-RealSense commented 1 month ago

Hi @edgarcamilocamacho Do you have an update about this case that you can provide, please? Thanks!

vahapt commented 1 month ago

Hi @MartyG-RealSense, I've replicated the same problem on the following hardware setups.

Configuration Setup 1 Setup 2
Camera Model D435 D435
Firmware Version 5.16.0.1 5.16.0.1
Operating System & Version Ubuntu v22.04 aarch64 Virtualized via VMWare under MacOS (M3 processor) [USB passthrough is enabled] Ubuntu v22.04 aarch64
Kernel Version (Linux Only) 6.8.0-45 6.1.43
Platform PC aarch64 Orange PI 5 - rockchip - aarch64
Librealsense SDK Version 2.55.1 2.55.1
Language C++ C++
Segment Robot Robot
ROS Distro Humble Humble
RealSense ROS Wrapper Version v4.55.1 v4.55.1
I've the following findings (Using Setup 1 hardware): In short: ROS2 Repos Librealsense Repos Built from Source
librealsense2.so Works Does not Work Works
librealsense2-gl.so N/A Does not Work Cannot be built

Long story:

Scenario1:

Scenario2:

Note the following: now -gl version is linked against the library built by librealsense, but non-gl library is linked against the library built by ros

/opt/ros/humble/lib/aarch64-linux-gnu/librealsense2.so.2.55.1
/usr/lib/aarch64-linux-gnu/librealsense2-gl.so.2.55.1

When I run the node with accelerate_gpu_with_glsl:=True point cloud is not generated. Otherwise it is generated without problems because ros version of librealsense2 is running as expected.

Scenario 3:

Scenario4:

vahapt commented 1 month ago

Update: I've managed to compile librealsense2-gl.so with the following cmake options:

cmake ../ -DBUILD_EXAMPLES=true -DBUILD_GRAPHICAL_EXAMPLES=true -DBUILD_GLSL_EXTENSIONS=ON -DFORCE_RSUSB_BACKEND=true -DCMAKE_BUILD_TYPE=Release -DBUILD_TOOLS=true

We're back to square 1. Point cloud can now be generated with CPU but not with GPU. On another note, (I'm not sure if it's directly related to this case) example app rs-gl works with both CPU and GPU and effectively reduces CPU load when GPU is enabled.

MartyG-RealSense commented 1 month ago

control_transfer returned warnings can be safely disregarded if the pointcloud appears to be performing normally.

vahapt commented 3 weeks ago

I've found the reason and implemented a solution, I'll test and let you know about the results.

The problem occurs in "librealsense" library if point cloud generation is opengl accelerated and there's no depth_texture provided. depth_to_points function returns empty point data as this operation is deferred to get_texture_map function. However, since there's no depth texture provided, get_texture_map is never called and we get empty point cloud. (see librealsense/pointclud.cpp@process_depth_frame function, if (_extrinsics && _other_intrinsics) is always false)

I've created another shader which just does point cloud extraction but not depth texture mapping and executed it within depth_to_points function if texture map is not available.

(See also https://github.com/resdocta/librealsense/commit/81b78a11490e763dc360454fd6b4f6a87723133a for details)

MartyG-RealSense commented 3 weeks ago

Thanks so much for sharing the details of your solution. I look forward to hearing about your test results. :)

vahapt commented 3 weeks ago

I've tested it successfully on OrangePI 5, GPU utilization is around 9% and CPU utilization is around 7% with depth camera running at 848x480@90FPS.

I should note that CPU utilization increases and frame rate drops to around 20FPS when subscribed to the ros2 topic (possibly due to message copying). But this is out of scope of this conversation.

Pull request to the development branch: https://github.com/IntelRealSense/librealsense/pull/13462

MartyG-RealSense commented 3 weeks ago

Thanks again for creating the PR! :)

edgarcamilocamacho commented 3 days ago

Hello guys, I cherry picked the changes to the last version of librealsense and it's working, so I guess I'd wait for the new version with the change applied to test it again and close the issue.

MartyG-RealSense commented 2 days ago

Thanks very much @edgarcamilocamacho for the update - I'm pleased to hear that you achieved a working installation.