search

stereolabs / zed-ros2-wrapper

ROS 2 wrapper for the ZED SDK
https://www.stereolabs.com/docs/ros2/
Apache License 2.0
140 stars 144 forks source link

Error opening camera: NO GPU DETECTED #136

Closed lucyannofrota closed 1 year ago

lucyannofrota commented 1 year ago

Preliminary Checks

Description

I'm trying to use the ZED2 camera in a docker container. The docker container has ROS2 Foxy installed and the ZED SDK. The ZED wrapper was built without a problem.

I got the error: "Error opening camera: NO GPU DETECTED" when I try to run ros2 launch zed_wrapper zed2.launch.py.

Steps to Reproduce

  1. Run the container

    xhost +si:localuser:root docker run --gpus all -it --privileged -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix stereolabs/zed:3.8-gl-devel-cuda11.4-ubuntu20.04

  2. ros2 launch zed_wrapper zed2.launch.py

Expected Result

Being able to launch the node

Actual Result

ros2 launch zed_wrapper zed2.launch.py

[INFO] [launch]: All log files can be found below /home/ros/.ros/log/2023-02-20-20-01-38-583565-ISR-Desktop-453653 [INFO] [launch]: Default logging verbosity is set to INFO [INFO] [robot_state_publisher-1]: process started with pid [453663] [INFO] [zed_wrapper-2]: process started with pid [453665] [robot_state_publisher-1] Parsing robot urdf xml string. [robot_state_publisher-1] Link zed2_base_link had 1 children [robot_state_publisher-1] Link zed2_camera_center had 4 children [robot_state_publisher-1] Link zed2_baro_link had 0 children [robot_state_publisher-1] Link zed2_left_camera_frame had 2 children [robot_state_publisher-1] Link zed2_left_camera_optical_frame had 0 children [robot_state_publisher-1] Link zed2_temp_left_link had 0 children [robot_state_publisher-1] Link zed2_mag_link had 0 children [robot_state_publisher-1] Link zed2_right_camera_frame had 2 children [robot_state_publisher-1] Link zed2_right_camera_optical_frame had 0 children [robot_state_publisher-1] Link zed2_temp_right_link had 0 children [robot_state_publisher-1] [INFO] [1676923298.810269616] [zed2.zed_state_publisher]: got segment base_link [robot_state_publisher-1] [INFO] [1676923298.810340142] [zed2.zed_state_publisher]: got segment zed2_baro_link [robot_state_publisher-1] [INFO] [1676923298.810350758] [zed2.zed_state_publisher]: got segment zed2_base_link [robot_state_publisher-1] [INFO] [1676923298.810358424] [zed2.zed_state_publisher]: got segment zed2_camera_center [robot_state_publisher-1] [INFO] [1676923298.810365561] [zed2.zed_state_publisher]: got segment zed2_left_camera_frame [robot_state_publisher-1] [INFO] [1676923298.810372624] [zed2.zed_state_publisher]: got segment zed2_left_camera_optical_frame [robot_state_publisher-1] [INFO] [1676923298.810379578] [zed2.zed_state_publisher]: got segment zed2_mag_link [robot_state_publisher-1] [INFO] [1676923298.810386440] [zed2.zed_state_publisher]: got segment zed2_right_camera_frame [robot_state_publisher-1] [INFO] [1676923298.810393285] [zed2.zed_state_publisher]: got segment zed2_right_camera_optical_frame [robot_state_publisher-1] [INFO] [1676923298.810400098] [zed2.zed_state_publisher]: got segment zed2_temp_left_link [robot_state_publisher-1] [INFO] [1676923298.810406934] [zed2.zed_state_publisher]: got segment zed2_temp_right_link

[zed_wrapper-2] [INFO] [1676923298.824029919] [zed2.zed_node]: ZED Camera Component

[zed_wrapper-2] [INFO] [1676923298.824050966] [zed2.zed_node]: namespace: /zed2 [zed_wrapper-2] [INFO] [1676923298.824059983] [zed2.zed_node]: node name: zed_node

[zed_wrapper-2] [INFO] [1676923298.824085818] [zed2.zed_node]: DEBUG parameters [zed_wrapper-2] [INFO] [1676923298.824114016] [zed2.zed_node]: Debug mode: FALSE [zed_wrapper-2] [INFO] [1676923298.824138156] [zed2.zed_node]: Debug sensors: FALSE [zed_wrapper-2] [INFO] [1676923298.824150443] [zed2.zed_node]: GENERAL parameters [zed_wrapper-2] [INFO] [1676923298.824178806] [zed2.zed_node]: Camera model: zed2 - ZED 2 [zed_wrapper-2] [INFO] [1676923298.824206457] [zed2.zed_node]: SDK Verbose: 1 [zed_wrapper-2] [INFO] [1676923298.824230790] [zed2.zed_node]: SVO: live [zed_wrapper-2] [INFO] [1676923298.824249800] [zed2.zed_node]: SVO Loop: FALSE [zed_wrapper-2] [INFO] [1676923298.824266142] [zed2.zed_node]: SVO Realtime: TRUE [zed_wrapper-2] [INFO] [1676923298.824283309] [zed2.zed_node]: Camera name: zed2 [zed_wrapper-2] [INFO] [1676923298.824305322] [zed2.zed_node]: Camera ID: 0 [zed_wrapper-2] [INFO] [1676923298.824323625] [zed2.zed_node]: Camera SN: 0

[zed_wrapper-2] [INFO] [1676923298.824365949] [zed2.zed_node]: Camera reconnection temptatives: 5 [zed_wrapper-2] [INFO] [1676923298.824398422] [zed2.zed_node]: Camera framerate: 15 [zed_wrapper-2] [INFO] [1676923298.824427591] [zed2.zed_node]: GPU ID: -1 [zed_wrapper-2] [INFO] [1676923298.824460150] [zed2.zed_node]: Camera resolution: HD720 [zed_wrapper-2] [INFO] [1676923298.824493553] [zed2.zed_node]: Publishing resolution: MEDIUM [zed_wrapper-2] [INFO] [1676923298.824541962] [zed2.zed_node]: Region of interest: [zed_wrapper-2] [INFO] [1676923298.824572842] [zed2.zed_node]: Camera self calibration: TRUE [zed_wrapper-2] [INFO] [1676923298.824594261] [zed2.zed_node]: Camera flip: FALSE

[zed_wrapper-2] [INFO] [1676923298.824646140] [zed2.zed_node]: VIDEO parameters [zed_wrapper-2] [INFO] [1676923298.824670930] [zed2.zed_node]: Use old extrinsic parameters: 0 [zed_wrapper-2] [INFO] [1676923298.824700691] [zed2.zed_node]: [DYN] Brightness: 4 [zed_wrapper-2] [INFO] [1676923298.824729581] [zed2.zed_node]: [DYN] Contrast: 4 [zed_wrapper-2] [INFO] [1676923298.824757239] [zed2.zed_node]: [DYN] Hue: 0 [zed_wrapper-2] [INFO] [1676923298.824791494] [zed2.zed_node]: [DYN] Saturation: 4 [zed_wrapper-2] [INFO] [1676923298.824822380] [zed2.zed_node]: [DYN] Sharpness: 4 [zed_wrapper-2] [INFO] [1676923298.824842339] [zed2.zed_node]: [DYN] Gamma: 8 [zed_wrapper-2] [INFO] [1676923298.824859750] [zed2.zed_node]: [DYN] Auto Exposure/Gain: TRUE [zed_wrapper-2] [INFO] [1676923298.824876943] [zed2.zed_node]: [DYN] Exposure: 80 [zed_wrapper-2] [INFO] [1676923298.824897490] [zed2.zed_node]: [DYN] Gain: 80 [zed_wrapper-2] [INFO] [1676923298.824914593] [zed2.zed_node]: [DYN] Auto White Balance: TRUE [zed_wrapper-2] [INFO] [1676923298.824932238] [zed2.zed_node]: [DYN] White Balance Temperature: 42 [zed_wrapper-2] [INFO] [1676923298.824951825] [zed2.zed_node]: Video QoS History: KEEP_LAST [zed_wrapper-2] [INFO] [1676923298.824977247] [zed2.zed_node]: Video QoS History depth: 1 [zed_wrapper-2] [INFO] [1676923298.824994948] [zed2.zed_node]: Video QoS Reliability: RELIABLE [zed_wrapper-2] [INFO] [1676923298.825011490] [zed2.zed_node]: Video QoS Durability: VOLATILE [zed_wrapper-2] [INFO] [1676923298.825022784] [zed2.zed_node]: DEPTH parameters [zed_wrapper-2] [INFO] [1676923298.825042213] [zed2.zed_node]: * Depth quality: 1 - PERFORMANCE

[zed_wrapper-2] [INFO] [1676923298.825107068] [zed2.zed_node]: Depth Sensing Mode: 0 - STANDARD [zed_wrapper-2] [INFO] [1676923298.825124487] [zed2.zed_node]: Depth Stabilization: TRUE [zed_wrapper-2] [INFO] [1676923298.825142512] [zed2.zed_node]: * OpenNI mode (16bit point cloud): FALSE

[zed_wrapper-2] [INFO] [1676923298.825180427] [zed2.zed_node]: [DYN] Depth Confidence: 50 [zed_wrapper-2] [INFO] [1676923298.825198583] [zed2.zed_node]: [DYN] Depth Texture Confidence: 100 [zed_wrapper-2] [INFO] [1676923298.825217526] [zed2.zed_node]: [DYN] Remove saturated areas: TRUE [zed_wrapper-2] [INFO] [1676923298.825266465] [zed2.zed_node]: Depth QoS History: KEEP_LAST [zed_wrapper-2] [INFO] [1676923298.825290335] [zed2.zed_node]: Depth QoS History depth: 1 [zed_wrapper-2] [INFO] [1676923298.825317882] [zed2.zed_node]: Depth QoS Reliability: RELIABLE [zed_wrapper-2] [INFO] [1676923298.825343167] [zed2.zed_node]: * Depth QoS Durability: VOLATILE [zed_wrapper-2] [INFO] [1676923298.825355134] [zed2.zed_node]: POSITIONAL TRACKING parameters [zed_wrapper-2] [INFO] [1676923298.825376172] [zed2.zed_node]: Positional tracking enabled: TRUE [zed_wrapper-2] [INFO] [1676923298.825397414] [zed2.zed_node]: Base frame id: base_link [zed_wrapper-2] [INFO] [1676923298.825419733] [zed2.zed_node]: Map frame id: map [zed_wrapper-2] [INFO] [1676923298.825443656] [zed2.zed_node]: Odometry frame id: odom [zed_wrapper-2] [INFO] [1676923298.825464926] [zed2.zed_node]: Broadcast Odometry TF: TRUE [zed_wrapper-2] [INFO] [1676923298.825486084] [zed2.zed_node]: Broadcast Pose TF: TRUE

[zed_wrapper-2] [INFO] [1676923298.825532049] [zed2.zed_node]: [DYN] Depth minimum range: 0 [zed_wrapper-2] [INFO] [1676923298.825557426] [zed2.zed_node]: [DYN] TF timestamp offset: 0 [zed_wrapper-2] [INFO] [1676923298.825589509] [zed2.zed_node]: [DYN] Path publishing rate: 2 [zed_wrapper-2] [INFO] [1676923298.825621405] [zed2.zed_node]: Path history lenght: -1 [zed_wrapper-2] [INFO] [1676923298.825658958] [zed2.zed_node]: Initial pose: [0,0,0,0,0,0,] [zed_wrapper-2] [INFO] [1676923298.825698458] [zed2.zed_node]: Area Memory: TRUE [zed_wrapper-2] [INFO] [1676923298.825730501] [zed2.zed_node]: Area Memory DB: [zed_wrapper-2] [INFO] [1676923298.825762343] [zed2.zed_node]: Camera is static: FALSE

[zed_wrapper-2] [INFO] [1676923298.825851364] [zed2.zed_node]: Floor Alignment: FALSE [zed_wrapper-2] [INFO] [1676923298.825870038] [zed2.zed_node]: Init Odometry with first valid pose data: TRUE [zed_wrapper-2] [INFO] [1676923298.825891517] [zed2.zed_node]: 2D mode: FALSE [zed_wrapper-2] [INFO] [1676923298.825908706] [zed2.zed_node]: Pose/Odometry QoS History: KEEP_LAST [zed_wrapper-2] [INFO] [1676923298.825924477] [zed2.zed_node]: Pose/Odometry QoS History depth: 1 [zed_wrapper-2] [INFO] [1676923298.825941538] [zed2.zed_node]: Pose/Odometry QoS Reliability: RELIABLE [zed_wrapper-2] [INFO] [1676923298.825973421] [zed2.zed_node]: * Pose/Odometry QoS Durability: VOLATILE [zed_wrapper-2] [INFO] [1676923298.825994642] [zed2.zed_node]: SENSORS STACK parameters [zed_wrapper-2] [INFO] [1676923298.826026680] [zed2.zed_node]: Sensors Camera Sync: FALSE [zed_wrapper-2] [INFO] [1676923298.826063617] [zed2.zed_node]: Sensors publishing rate: 200 Hz [zed_wrapper-2] [INFO] [1676923298.826099731] [zed2.zed_node]: Sensors QoS History: KEEP_LAST [zed_wrapper-2] [INFO] [1676923298.826130406] [zed2.zed_node]: Sensors QoS History depth: 1 [zed_wrapper-2] [INFO] [1676923298.826161802] [zed2.zed_node]: Sensors QoS Reliability: RELIABLE [zed_wrapper-2] [INFO] [1676923298.826191922] [zed2.zed_node]: Sensors QoS Durability: VOLATILE [zed_wrapper-2] [INFO] [1676923298.826229822] [zed2.zed_node]: Spatial Mapping parameters [zed_wrapper-2] [INFO] [1676923298.826266048] [zed2.zed_node]: * Spatial Mapping Enabled: FALSE

[zed_wrapper-2] [INFO] [1676923298.826426421] [zed2.zed_node]: Clicked point topic: /clicked_point [zed_wrapper-2] [INFO] [1676923298.826458697] [zed2.zed_node]: Sensors QoS History: KEEP_LAST [zed_wrapper-2] [INFO] [1676923298.826489268] [zed2.zed_node]: Sensors QoS History depth: 1 [zed_wrapper-2] [INFO] [1676923298.826520157] [zed2.zed_node]: Sensors QoS Reliability: RELIABLE [zed_wrapper-2] [INFO] [1676923298.826568690] [zed2.zed_node]: * Sensors QoS Durability: VOLATILE [zed_wrapper-2] [INFO] [1676923298.826599713] [zed2.zed_node]: OBJECT DETECTION parameters [zed_wrapper-2] [INFO] [1676923298.826632856] [zed2.zed_node]: Object Detection enabled: FALSE [zed_wrapper-2] [INFO] [1676923298.826675161] [zed2.zed_node]: OD min. confidence: 50

[zed_wrapper-2] [INFO] [1676923298.826972231] [zed2.zed_node]: OD tracking: TRUE [zed_wrapper-2] [INFO] [1676923298.827046747] [zed2.zed_node]: Object Detection model: 0 - MULTI CLASS BOX FAST [zed_wrapper-2] [INFO] [1676923298.827091715] [zed2.zed_node]: Object Filtering mode: 1 - NMS 3D [zed_wrapper-2] [INFO] [1676923298.827127952] [zed2.zed_node]: MultiClassBox people: TRUE [zed_wrapper-2] [INFO] [1676923298.827163310] [zed2.zed_node]: MultiClassBox vehicles: TRUE [zed_wrapper-2] [INFO] [1676923298.827200526] [zed2.zed_node]: MultiClassBox bags: TRUE [zed_wrapper-2] [INFO] [1676923298.827230361] [zed2.zed_node]: MultiClassBox animals: TRUE [zed_wrapper-2] [INFO] [1676923298.827265553] [zed2.zed_node]: MultiClassBox electronics: TRUE [zed_wrapper-2] [INFO] [1676923298.827301162] [zed2.zed_node]: MultiClassBox fruits and vegetables: TRUE [zed_wrapper-2] [INFO] [1676923298.827341283] [zed2.zed_node]: MultiClassBox sport-related objects: TRUE [zed_wrapper-2] [INFO] [1676923298.827378407] [zed2.zed_node]: Body format: 1 - POSE_34 [zed_wrapper-2] [INFO] [1676923298.827398239] [zed2.zed_node]: Skeleton fitting: TRUE (forced by object_detection.body_format) [zed_wrapper-2] [INFO] [1676923298.827419670] [zed2.zed_node]: Obj. Det. QoS History: KEEP_LAST [zed_wrapper-2] [INFO] [1676923298.827436392] [zed2.zed_node]: Obj. Det. QoS History depth: 1 [zed_wrapper-2] [INFO] [1676923298.827455392] [zed2.zed_node]: Obj. Det. QoS Reliability: RELIABLE [zed_wrapper-2] [INFO] [1676923298.827483499] [zed2.zed_node]: Obj. Det. QoS Durability: VOLATILE [zed_wrapper-2] [INFO] [1676923298.827573202] [zed2.zed_node]: SERVICES

[zed_wrapper-2] [INFO] [1676923298.831484073] [zed2.zed_node]: STARTING CAMERA [zed_wrapper-2] [INFO] [1676923298.831508028] [zed2.zed_node]: SDK Version: 3.8.2 - Build 63211_622b02a5 [zed_wrapper-2] [INFO] [1676923298.833911852] [zed2.zed_node]: CAMERA OPENING [zed_wrapper-2] [ZED][INFO] Logging level INFO [zed_wrapper-2] in void sl::Mat::alloc(size_t, size_t, sl::MAT_TYPE, sl::MEM) : Err [100]: no CUDA-capable device is detected. [zed_wrapper-2] [WARN] [1676923299.491832752] [zed2.zed_node]: Error opening camera: NO GPU DETECTED [zed_wrapper-2] [INFO] [1676923299.491890886] [zed2.zed_node]: Please verify the USB3 connection [zed_wrapper-2] No NVIDIA graphics card detected. Install an NVIDIA GPU, CUDA and restart your computer after completing installation. ^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)

[INFO] [robot_state_publisher-1]: process has finished cleanly [pid 453663] [zed_wrapper-2] CUDA error at Camera.cpp:121 code=100(cudaErrorNoDevice) "void sl::Camera::close()" [zed_wrapper-2] CUDA error at Camera.cpp:149 code=100(cudaErrorNoDevice) "void sl::Camera::close()" [zed_wrapper-2] CUDA error at Camera.cpp:174 code=100(cudaErrorNoDevice) "void sl::Camera::close()" [zed_wrapper-2] CUDA error at Camera.cpp:214 code=100(cudaErrorNoDevice) "void sl::Camera::close()" [zed_wrapper-2] CUDA error at Camera.cpp:219 code=100(cudaErrorNoDevice) "void sl::Camera::close()" [zed_wrapper-2] CUDA error at CameraUtils.hpp:773 code=100(cudaErrorNoDevice) "void sl::ObjectsDetectorHandler::clear()" [zed_wrapper-2] CUDA error at CameraUtils.hpp:789 code=100(cudaErrorNoDevice) "void sl::ObjectsDetectorHandler::clear()" [zed_wrapper-2] CUDA error at CameraUtils.hpp:795 code=100(cudaErrorNoDevice) "void sl::ObjectsDetectorHandler::clear()" [zed_wrapper-2] CUDA error at CameraUtils.hpp:798 code=100(cudaErrorNoDevice) "void sl::ObjectsDetectorHandler::clear()" [zed_wrapper-2] CUDA error at CameraUtils.hpp:800 code=100(cudaErrorNoDevice) "void sl::ObjectsDetectorHandler::clear()" [zed_wrapper-2] CUDA error at CameraUtils.hpp:809 code=100(cudaErrorNoDevice) "void sl::ObjectsDetectorHandler::clear()" [zed_wrapper-2] CUDA error at Camera.cpp:234 code=100(cudaErrorNoDevice) "void sl::Camera::close()" [zed_wrapper-2] [ZED][INFO] Logging level INFO [zed_wrapper-2] in void sl::Mat::alloc(size_t, size_t, sl::MAT_TYPE, sl::MEM) : Err [100]: no CUDA-capable device is detected. [zed_wrapper-2] No NVIDIA graphics card detected. Install an NVIDIA GPU, CUDA and restart your computer after completing installation. [zed_wrapper-2] [WARN] [1676923302.560481240] [zed2.zed_node]: Error opening camera: NO GPU DETECTED [zed_wrapper-2] [INFO] [1676923302.560494229] [zed2.zed_node]: Please verify the USB3 connection

ZED Camera model

ZED2

Environment

OS: Ubuntu 20.04
CPU: Intel(R) Core(TM) i7-3770
GPU: 2x GTX 780

Docker: 23.0.1
ROS: Foxy

NVIDIA-SMI 470.161.03   Driver Version: 470.161.03   CUDA Version: 11.7

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:49:14_PDT_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0

Anything else?

Inside the container, I was able to execute applications using the GPU like OpengGL apps.

Myzhar commented 1 year ago

Hi @lucyannofrota if the dockerfile used to build the image does not enable the GPU driver capabilities like here https://github.com/stereolabs/zed-ros2-wrapper/blob/master/docker/Dockerfile.u22-cu117-humble-devel you must enable them in the docker run command: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/user-guide.html#driver-capabilities

You can try to enable them all

lucyannofrota commented 1 year ago

Thank you for your reply @Myzhar. There was confusion in the information that I presented earlier. I'm not using the container stereolabs/zed:3.8-gl-devel-cuda11.4-ubuntu20.04 this container doesn't even have ROS installed! Sorry about that.

I'm actually running a custom container based on nvidia/cuda:11.7.0-devel-ubuntu20.04.

I've installed the ZED dependencies based on: https://github.com/stereolabs/zed-docker/blob/master/3.X/ubuntu/devel/Dockerfile.

The ZED SDK part of the dockerfile looks like this:

ENV NVIDIA_VISIBLE_DEVICES all

# ENV NVIDIA_DRIVER_CAPABILITIES \ # ${NVIDIA_DRIVER_CAPABILITIES:+$NVIDIA_DRIVER_CAPABILITIES,}compute,video,utility,graphics

ENV NVIDIA_DRIVER_CAPABILITIES all

RUN echo "Europe/Paris" > /etc/localtime ; echo "CUDA Version 11.7" > /usr/local/cuda/version.txt

USER ros ENV USER=ros # Install zstd RUN sudo apt-get update -y || true RUN sudo apt-get install --no-install-recommends zstd

# Setup the ZED SDK RUN sudo apt-get update -y || true RUN sudo apt-get install --no-install-recommends lsb-release wget less udev sudo build-essential cmake -y RUN sudo wget -q -O ZED_SDK_38_Linux_Ubuntu20.run https://download.stereolabs.com/zedsdk/3.8/cu117/ubuntu20 RUN sudo chmod +x ZED_SDK_38_Linux_Ubuntu20.run

RUN ./ZED_SDK_38_Linux_Ubuntu20.run -- silent

RUN sudo ln -sf /lib/x86_64-linux-gnu/libusb-1.0.so.0 /usr/lib/x86_64-linux-gnu/libusb-1.0.so \ && sudo rm ZED_SDK_38_Linux_Ubuntu20.run \ && sudo rm -rf /var/lib/apt/lists/*

FROM zed-sdk as zed-api

# # ZED Python API RUN sudo apt-get update -y || true RUN sudo apt-get install --no-install-recommends python3 python3-pip libpng-dev libgomp1 -y && \ wget download.stereolabs.com/zedsdk/pyzed -O /usr/local/zed/get_python_api.py && \ python3 /usr/local/zed/get_python_api.py && \ python3 -m pip install numpy opencv-python pyopengl .whl && \ sudo rm .whl ; sudo rm -rf /var/lib/apt/lists/*

# # Make some tools happy RUN sudo mkdir -p /root/Documents/ZED/

# ROS diagnostic_updater RUN sudo apt-get update -y || true RUN sudo apt-get install --no-install-recommends \ ros-foxy-diagnostic-updater

Extra notes:

I already had the variables NVIDIA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES set as you suggested.

I don't think the problem is related to those variables because I was able to run Gazebo with OpenGL and GPU support.

Myzhar commented 1 year ago

Hi @lucyannofrota I suggest you compare your dockerfile with this: https://github.com/stereolabs/zed-ros2-wrapper/blob/master/docker/Dockerfile.u22-cu117-humble-release available in the docker folder of this repository. It's highly optimized and it contains the latest updates.

You can easily modify it for Ubuntu 20 and Foxy if you need it

lucyannofrota commented 1 year ago

I tried to use the suggested container as it is as well as the altered version (Ubuntu 20.04 + ROS Foxy) and got the same error in both cases! "https://github.com/stereolabs/zed-ros2-wrapper/blob/master/docker/Dockerfile.u22-cu117-humble-release"

[zed_wrapper-2] [WARN] [1677091650.892548286] [zed2.zed_node]: Error opening camera: NO GPU DETECTED [zed_wrapper-2] [INFO] [1677091650.892599583] [zed2.zed_node]: Please verify the USB3 connection

nvidia-smi works and lists my two GPUs!

Myzhar commented 1 year ago

The NVIDIA driver version seems old, have you tried upgrading it?

lucyannofrota commented 1 year ago

I have two GTX 780. I did the tests mentioned above using version 470 of the nvidia drivers and had the cuda 11.4 installed. I don't know if this can be the source of the problem but I'm gonna try to update the driver to 515 and install cuda 11.7.

It's possible that this can be a problem with the camera connection?

Myzhar commented 1 year ago

Let me know if by updating the driver you fix the issue. I have also a concern regarding the double GPU configuration. Normally the ZED SDK takes the first available, but maybe you need to manually force it by setting the general.gpu_id parameter to 0 or 1: https://github.com/stereolabs/zed-ros2-wrapper/blob/master/zed_wrapper/config/common.yaml#L22

lucyannofrota commented 1 year ago

I was not able to install the new drivers! 470 is the last NVIDIA driver that supports GTX780, consequently, the last cuda available is 11.4.

I've tested the container with NVIDIA_VISIBLE_DEVICES=0 as well as setting general.gpu_id = 0 or general.gpu_id = 1 and I got the same error.

I think I just have found the source of the issue. The graphics cards + drivers + Nvidia-container-runtime are working as intended! But apparently exists an incompatibility of ZED SDK 3.8 with Maxwell and Kepler GPUs. This information is listed as a known issue ("Maxwell and Kepler GPUs (compute capabilities 35, 50 and 52) are not supported using CUDA 11.X, only with CUDA 10.2") in (https://www.stereolabs.com/developers/release/).

Considering this. I think I have two options:

  1. Try ZED SDK 3.6
  2. Downgrade to CUDA 10.2

What do you recommend? I'm inclined to try the first option "ZED SDK 3.6". There's a major loss using ZED SDK 3.6 instead of ZED SDK 3.8? I'm working with ROS Foxy and I have two setups. One desktop for development (Ubuntu 20.04 + 2xGTX780) and a Jetson AGX Xavier with JetPack 5.1 (L4T R35.2.1) for deployment.

lucyannofrota commented 1 year ago

I just realized that the current version of "https://github.com/stereolabs/zed-ros2-wrapper" only supports ZED SDK 3.8. There's an older version that supports the ZED SDK 3.6?

Myzhar commented 1 year ago

You can get the wrapper for SDK v3.6 in the releases

https://github.com/stereolabs/zed-ros2-wrapper/releases/tag/foxy-v3.6

lucyannofrota commented 1 year ago

The SDK v3.6 didn't work as well!

Myzhar commented 1 year ago

I expected that, in my opinion your problem is related to an incompatibility issue between the NVIDIA docker and the GPU driver.

What is the result of this command? sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.7.0-devel-ubuntu20.04 nvidia-smi

Are you able to run one of the CUDA examples inside a Docker container?

lucyannofrota commented 1 year ago

Yes. I can run this command! All the previous nvidia-smi provided before were from inside containers.

sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.7.0-devel-ubuntu20.04 nvidia-smi output:

lucyanno@ISR-Desktop:~/Semantic-Mapping-ROS2-Containers$ sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.7.0-devel-ubuntu20.04 nvidia-smi

==========
== CUDA ==
==========

CUDA Version 11.7.0

Container image Copyright (c) 2016-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

*************************
** DEPRECATION NOTICE! **
*************************
THIS IMAGE IS DEPRECATED and is scheduled for DELETION.
    https://gitlab.com/nvidia/container-images/cuda/blob/master/doc/support-policy.md

Fri Feb 24 10:28:41 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.161.03   Driver Version: 470.161.03   CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 N/A |                  N/A |
| 37%   29C    P8    N/A /  N/A |    503MiB /  3021MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:02:00.0 N/A |                  N/A |
| 37%   24C    P8    N/A /  N/A |     12MiB /  3022MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
lucyannofrota commented 1 year ago

I did tryied to run the CUDA examples at "https://github.com/NVIDIA/cuda-samples/tree/v11.6".

I managed to compile the samples. But I couldn't execute than ("No device detected"). So the problem is related to the NVIDIA runtime!

Thank you verry mutch for your help @Myzhar but after that I don't think this problem is related with the ZED SDK at all. It should be a CUDA + NVIDIA driver compatibility issue with this specific achitecture. So the solution should be downgrade to CUDA 10.2 but it isn't supported by ubuntu 20.04.

Myzhar commented 1 year ago

OK, let me know if you find a solution

lucyannofrota commented 1 year ago

I ended up switching my graphics card and it worked perfectly with the GTX1060. I think we can close this topic knowing that the problem is related to an incompatibility of CUDA 11.4 with Maxwell and Keppler cards. Thanks for your help @Myzhar.