ROS2 Foxy Docker Pointcloud Display Black

nfs-chris commented 1 year ago

Preliminary Checks

[X] This issue is not a duplicate. Before opening a new issue, please search existing issues.
[X] This issue is not a question, feature request, or anything other than a bug report directly related to this project.

Description

When starting the Zed Mini camera from within Docker the published pointcloud returns a black image when visualized. Running ros2 topic hz /zedm/zed_node/point_cloud/cloud_registered shows that data is being sent:

root@3c1f21d12482:/app# ros2 topic hz /zedm/zed_node/point_cloud/cloud_registered
average rate: 10.002
    min: 0.094s max: 0.105s std dev: 0.00344s window: 11
average rate: 9.990
    min: 0.092s max: 0.109s std dev: 0.00372s window: 21
average rate: 9.987
    min: 0.092s max: 0.109s std dev: 0.00347s window: 31
average rate: 9.989
    min: 0.092s max: 0.109s std dev: 0.00343s window: 42
average rate: 10.002
    min: 0.092s max: 0.109s std dev: 0.00347s window: 53

and other topics, such as zedm/zed_node/rgb_raw/image_raw_color, work as intended. Setting QoS on my (python) subscriber, as mentioned here, in the following way

from rclpy.qos import QoSProfile, QoSReliabilityPolicy, QoSHistoryPolicy, DurabilityPolicy

self.qos_profile = QoSProfile(
    history=QoSHistoryPolicy.RMW_QOS_POLICY_HISTORY_KEEP_LAST,
    depth=1,
    reliability=QoSReliabilityPolicy.RMW_QOS_POLICY_RELIABILITY_RELIABLE,
    durability=DurabilityPolicy.RMW_QOS_POLICY_DURABILITY_VOLATILE
)

self.pcd_subscriber = self.create_subscription(
    PointCloud2,
    'zedm/zed_node/point_cloud/cloud_registered',
    self.listener_callback,
    self.qos_profile
)

does not help. Installing version v3.0.0 of theimage_transport package via:

git clone https://github.com/ros-perception/image_common.git --branch 3.0.0 --single-branch

also didn't fix this and I have tried different lighting and distance settings.

My issue seems related to this discussion, which appears to be stale and I have not been able to find a solution. Any help would be greatly appreciated.

Steps to Reproduce

Build ROS2 Foxy Docker
Start Docker which will start the camera via ros2 launch zed_wrapper zedm.launch.py
Subscribe to /zedm/zed_node/point_cloud/cloud_registered
Visualize the PointCloud2 data (I use open3d)

Expected Result

Pointcloud being visualised correctly.

Actual Result

Black image with following data preview when inspecting topic:

root@7c0d23a6a14d:/app# ros2 topic echo /zedm/zed_node/point_cloud/cloud_registered
header:
  stamp:
    sec: 1670335559
    nanosec: 560916346
  frame_id: zedm_left_camera_frame
height: 720
width: 1280
fields:
- name: x
  offset: 0
  datatype: 7
  count: 1
- name: y
  offset: 4
  datatype: 7
  count: 1
- name: z
  offset: 8
  datatype: 7
  count: 1
- name: rgb
  offset: 12
  datatype: 7
  count: 1
is_bigendian: false
point_step: 16
row_step: 20480
data:
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- 255
- 255
- 255
- 127
- '...'
is_dense: false
---

ZED Camera model

ZED Mini

Environment

OS: Ubuntu 20.04
GPU: NVIDIA GeForce RTX 3090 Ti
_Docker ARGs_
UBUNTU_RELEASE_YEAR=20
ZED_SDK_MAJOR=3
ZED_SDK_MINOR=8
CUDA_MAJOR=11
CUDA_MINOR=4
ROS_DISTRO=foxy

Anything else?

I'm using a slightly modified Docker script, as the original one did not work and I'm passing the GPU via Docker Compose.

Dockerfile:

ARG UBUNTU_RELEASE_YEAR=20
ARG ZED_SDK_MAJOR=3
ARG ZED_SDK_MINOR=8
ARG CUDA_MAJOR=11
ARG CUDA_MINOR=4

FROM stereolabs/zed:${ZED_SDK_MAJOR}.${ZED_SDK_MINOR}-gl-devel-cuda${CUDA_MAJOR}.${CUDA_MINOR}-ubuntu${UBUNTU_RELEASE_YEAR}.04

ENV LOGNAME root
ENV ROS_DISTRO=foxy
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8

RUN apt-get update || true && apt-get install -q -y --no-install-recommends \
    bash-completion \
    dirmngr \
    gnupg2 \
    lsb-release \
    python3-pip \
    && rm -rf /var/lib/apt/lists/*

RUN echo "deb http://packages.ros.org/ros2/ubuntu focal main" > /etc/apt/sources.list.d/ros2-latest.list && \
    apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654

RUN apt-get update || true && apt-get install --no-install-recommends -y \
    build-essential \
    git \
    python3-colcon-common-extensions \
    python3-colcon-mixin \
    python3-rosdep \
    python3-vcstool \
    && rm -rf /var/lib/apt/lists/*

RUN apt-get update || true && apt-get install --no-install-recommends -y \
    ros-${ROS_DISTRO}-ros-base \
    ros-${ROS_DISTRO}-rclcpp   \
    ros-${ROS_DISTRO}-rclcpp-components   \
    ros-${ROS_DISTRO}-rcutils   \
    ros-${ROS_DISTRO}-builtin-interfaces   \
    ros-${ROS_DISTRO}-std-msgs   \
    ros-${ROS_DISTRO}-stereo-msgs   \
    ros-${ROS_DISTRO}-sensor-msgs   \
    ros-${ROS_DISTRO}-geometry-msgs   \
    ros-${ROS_DISTRO}-nav-msgs   \
    ros-${ROS_DISTRO}-tf2   \
    ros-${ROS_DISTRO}-tf2-ros   \
    ros-${ROS_DISTRO}-tf2-geometry-msgs   \
    ros-${ROS_DISTRO}-image-transport   \
    ros-${ROS_DISTRO}-std-srvs   \
    ros-${ROS_DISTRO}-diagnostic-msgs   \
    ros-${ROS_DISTRO}-diagnostic-updater   \
    ros-${ROS_DISTRO}-visualization-msgs   \
    ros-${ROS_DISTRO}-ament-lint-auto   \
    ros-${ROS_DISTRO}-ament-lint-common   \
    ros-${ROS_DISTRO}-ament-cmake-clang-format   \
    ros-${ROS_DISTRO}-launch-ros   \
    ros-${ROS_DISTRO}-ament-cmake   \
    ros-${ROS_DISTRO}-xacro   \
    ros-${ROS_DISTRO}-shape-msgs   \
    ros-${ROS_DISTRO}-rosidl-default-generators   \
    ros-${ROS_DISTRO}-rosidl-default-runtime   \
    && rm -rf /var/lib/apt/lists/*

RUN pip3 install -U \
    argcomplete \
    flake8 \
    flake8-blind-except \
    flake8-builtins \
    flake8-class-newline \
    flake8-comprehensions \
    flake8-deprecated \
    flake8-docstrings \
    flake8-import-order \
    flake8-quotes \
    pytest-repeat \
    pytest-rerunfailures

RUN colcon mixin add default \
    https://raw.githubusercontent.com/colcon/colcon-mixin-repository/master/index.yaml && \
    colcon mixin update && \
    colcon metadata add default \
    https://raw.githubusercontent.com/colcon/colcon-metadata-repository/master/index.yaml && \
    colcon metadata update

WORKDIR /app
COPY zed_mini/ros2_entrypoint.sh /app

RUN ["chmod", "+x", "ros2_entrypoint.sh"]
ENTRYPOINT ["./ros2_entrypoint.sh"]

with the following entrypoint:

#!/bin/bash

DIR=ros2_ws

source /opt/ros/foxy/setup.bash

if [ -d ~/$DIR ]; then
    echo "Starting Zed Mini..."
    cd ~/ros2_ws
else
    mkdir -p ~/ros2_ws/src
    cd ~/ros2_ws/src/
    git clone https://github.com/ros-perception/image_common.git --branch 3.0.0 --single-branch
    git clone --recursive https://github.com/stereolabs/zed-ros2-wrapper.git
    cd ~/ros2_ws
    rosdep init
    rosdep update
    rosdep install --from-paths src --ignore-src -r -y
    colcon build --symlink-install --cmake-args=-DCMAKE_BUILD_TYPE=Release
fi

source install/local_setup.bash

ros2 launch zed_wrapper zedm.launch.py

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

Myzhar commented 1 year ago

@nfs-chris We provided Dockerfile files in the new docker folder, can you try the release version (you can adapt it to Foxy and Ubuntu 20 if you need).

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

nfs-chris commented 1 year ago

Thank you for the docker files @Myzhar. Unfortunately, it seems the GPU is not picked up correctly:

[ZED][INFO] Logging level INFO
[zed_wrapper-2] in void sl::Mat::alloc(size_t, size_t, sl::MAT_TYPE, sl::MEM) : Err [804]: forward compatibility was attempted on non supported HW.
[zed_wrapper-2] No NVIDIA graphics card detected. Install an NVIDIA GPU, CUDA and restart your computer after completing installation.
[zed_wrapper-2] [WARN] [1675671033.258160341] [zedm.zed_node]: Error opening camera: NO GPU DETECTED

As other containers which are running on the same machine do detect the GPU, despite using the same parameters, there still seems to be an issue with how the image is built. I will investigate further and try out different parameters.

Myzhar commented 1 year ago

Are you using the correct commands to start the container? The host device must have a CUDA capable NVIDIA GPU, NVIDIA docker support must be correctly installed and finally the correct commands must be used to start the container. More info: https://www.stereolabs.com/docs/docker/

nfs-chris commented 1 year ago

I am using docker compose up zedm with the following docker-compose.yml:

zedm:
  build:
    context: .
    dockerfile: zed_mini.Dockerfile
  devices:
    - /dev/bus/usb:/dev/bus/usb
    - /dev/dri:/dev/dri
    - /dev/nvidia0:/dev/nvidia0
    - /dev/nvidiactl:/dev/nvidiactl
    - /dev/nvidia-uvm:/dev/nvidia-uvm
  privileged: true
  volumes:
    - /dev/shm:/dev/shm
    - /tmp/.X11-unix:/tmp/.X11-unix
    - /usr/lib/nvidia-470:/usr/lib/nvidia-470
  environment:
    - DISPLAY=$DISPLAY
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
             count: 1
             capabilities: [gpu]

To check if the GPU is available I run nvidia-smi before running the ros2 command to start the zed mini:

echo "Starting Zed Mini..."
# setup ros2 environment
source "/opt/ros/$ROS_DISTRO/setup.bash" --
source "/root/ros2_ws/install/local_setup.bash" --

# Welcome information
echo "ZED ROS2 Docker Image"
echo "---------------------"
echo 'ROS distro: ' $ROS_DISTRO
echo 'DDS middleware: ' $RMW_IMPLEMENTATION 
echo "---" 
echo 'Available ZED packages:'
ros2 pkg list | grep zed
echo "---------------------"

nvidia-smi
ros2 launch zed_wrapper zedm.launch.py

which gives the following output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.161.03   Driver Version: 470.161.03   CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  Off |
|  0%   32C    P8    19W / 450W |    384MiB / 24253MiB |      8%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

but then crashes with the previously mentioned error about no GPU being detected.

alassagne-sl commented 1 year ago

Hello, you may miss a few things in your compose file

services:
  your_awesome_service:
    runtime: nvidia
    environment:
      - NVIDIA_DRIVER_CAPABILITIES=all
    privileged: true

Can you try these ?

nfs-chris commented 1 year ago

Hi, I am following the official docs which can be found here, setting the runtime and driver capabilities accordingly (and this works for other setups I use). privileged: true is also set, as you can see from my previous post.

alassagne-sl commented 1 year ago

Can you update your nvidia driver ? it's quite old, maybe it's getting compatibility issues. 515 version (sudo apt install nvidia-driver-515) would do the trick.

nfs-chris commented 1 year ago

Unfortunately, there is other software running on the machine that only works with the 470 driver, but I will test out a newer driver when I find some time and will update this response, thanks.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

stereolabs / zed-ros2-wrapper