Zed 2i Multi-Camera Detection Issue

alberthli commented 11 months ago

Preliminary Checks

[X] This issue is not a duplicate. Before opening a new issue, please search existing issues.
[X] This issue is not a question, feature request, or anything other than a bug report directly related to this project.

Description

We're experiencing a bug where when using multiple ZED 2i cameras, the detection of the cameras is inconsistent. We are using a containerized setup via Docker. The relevant files to reproduce are attached here.

When running the multi-camera launch file provided in the Zed ROS2 examples repository with the command

ros2 launch zed_multi_camera zed_multi_camera.launch.py cam_names:="[cam1, cam2]" cam_models:="[zed2i, zed2i]" cam_serials:="[32144978,33143189]" cam_poses:="[[0.0,0.0,0.0,0.0,0.0,0.0], [0.0,0.0,0.0,0.0,0.0,0.0]]"

we see the following error:

[zed_wrapper-4] [ZED][WARNING] CAMERA NOT DETECTED in sl::ERROR_CODE sl::Camera::open(sl::InitParameters)
[zed_wrapper-4] [WARN] [1689024671.018911764] [camera_33143189.zed_node]: Error opening camera: CAMERA NOT DETECTED
[zed_wrapper-4] [INFO] [1689024671.018976626] [camera_33143189.zed_node]: Please verify the camera connection

Additional details regarding the error:

only one camera is not detected when two are plugged in
the error occurs randomly - sometimes, both are detected, while sometimes, one is not
when the error appears, we are not able to detect both again until unplugging and re-plugging the cameras, which often requires changing their ports. This also does not guarantee that the cameras will be detected without error
we can always detect the same single camera (as opposed to multiple) without error using the command ros2 launch zed_wrapper zed2i.launch.py

Additional things we have tested:

checking that the cameras are on different buses and that they appear in the output of lshw
checking before launching the multi-camera example that the cameras are detected using the Python API (in the REPL, we run import pyzed.sl as sl; sl.Camera.get_device_list())
using ROS2 Humble instead of Foxy

Steps to Reproduce

Because it is unclear what exactly causes the failure, we cannot reproduce the bug with 100% accuracy. However, doing something in the following list usually causes it.

Build the provided docker image using the command
```
docker image build -t test .
```
Start a container using the provided bash script with the command
```
bash startup.sh -n test
```
in the directory where the provided files are unzipped.

In the container, run the command (possibly multiple times until failure)

ros2 launch zed_multi_camera zed_multi_camera.launch.py cam_names:="[cam1, cam2]" cam_models:="[zed2i, zed2i]" cam_serials:="[32144978,33143189]" cam_poses:="[[0.0,0.0,0.0,0.0,0.0,0.0], [0.0,0.0,0.0,0.0,0.0,0.0]]"

If step 3 does not cause failure, try joining the existing container by opening a new one using the command
```
bash startup.sh -n test -j
```
Run the command from step 3 repeatedly.
If step 4 does not cause failure, try exiting the container from step 4 and rejoining. Again, run the command from step 3 until failure.
If step 5 does not cause failure, unplug and replug the cameras in the same or different port and repeat step 5.

Expected Result

We would expect that the two cameras should launch and be detected without issue.

Actual Result

However, only one is detected most of the time, and usually the issue comes up when we need to start/stop our Zed-related nodes.

ZED Camera model

ZED2i

Environment

OS: Ubuntu 22.04 on local machine, Ubuntu 20.04 in Docker container
CPU: AMD Ryzen Threadripper PRO 5995WX 64-Cores
GPUs: Nvidia RTX4090 (x2)
ZED SDK version: 4.0
ROS version: Foxy (also tested on Humble)

Output of nvcc --version:

Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0

Myzhar commented 11 months ago

Hi @alberthli what is the SDK version that you are using?

alberthli commented 11 months ago

We're using version 4.0. Specifically, we're using the Stereolabs base image stereolabs/zed:4.0-devel-cuda12.1-ubuntu20.04 from Dockerhub (file provided near the top of the issue).

Myzhar commented 11 months ago

This is a problem not related to the ZED ROS2 Wrapper, but to the ZED SDK. The ZED SDK team is already on it and a fix will be released as soon as possible with one of the next patch versions. Thank you for reporting and for the useful information. If you have other useful information please send an email to support@stereolabs.com. I close this issue.

alberthli commented 11 months ago

Thank you for the speedy reply. I'll keep an eye out for the next release. If the team needs more specific information, feel free to tag me on this issue as well, and I would be happy to provide it.

lzyang2000 commented 11 months ago

We are also running into the exact same problem - may I ask is there any update on the timeline of the patch? Thank you!

stereolabs / zed-ros2-wrapper