Unable to subscribe to color and depth from separate computer

[draketmoore]

---

Required Info
Camera Model	D405, D435i
Firmware Version	5.15.0.2
Operating System & Version	Ubuntu 22
Kernel Version (Linux Only)	(e.g. 5.4)
Platform	PC
Librealsense SDK Version	2.55.1
Language	python }
Segment	Robot
ROS Distro	Humble
RealSense ROS Wrapper Version	4.55.1

Issue Description

I currently have a dual computer setup, with a robot using an onboard NUC running the realsense-ros nodes, and a desktop subscribing to the color and depth topics. All works fine streaming and subscribing to either only the depth or color topics one at a time, but when I attempt to subscribe to both simultaneously, the depth stream stops entirely, even on the robot's side. This issue occurs even when attempting to view both streams in RVIZ on the remote desktop. I have no issues streaming both color and depth on the robot's onboard computer though. Any info/help would be greatly appreciated. Thank you!

[MartyG-RealSense]

Hi @draketmoore Are you able to stream both depth and color if you use a low resolution and FPS for both streams, such as 640x480 and 15 FPS?

ros2 launch realsense2_camera rs_launch.py depth_module.profile:=640x480x15 rgb_camera.color_profile:=640x480x15

[draketmoore]

Hi @MartyG-RealSense, thanks for your reply. I tried adjusting them to a very low setting of 480x270x15 already, but the issue remains.

[MartyG-RealSense]

As you are using a robot, is the communication between the robot's NUC and the desktop computer done by wireless wi-fi?

[draketmoore]

Yes, I have them both set up on the same wi-fi network, and have the ROS_DOMAIN ids set. I've tried using chrony in case there were any synchronization errors, as well as changing the DDS implementation to Cyclone, but nothing seems to be working. I have no trouble streaming other ROS topics from the robot, only the streams from the Realsenses.

[MartyG-RealSense]

If you enable the Infra1 topic by adding enable_infra1:=true to your launch and then subscribe to /camera/camera/infra1/image_rect_raw instead of color, are both infrared and depth able to both stream at the same time? If they are then it could indicate that the problem is with using color in a two-stream setup.

[draketmoore]

Hi @MartyG-RealSense, I just tested it out and I am able to subscribe to both infrared and depth simultaneously, but neither of those work alongside the color stream.

[MartyG-RealSense]

So the problem does seem to be with the color stream topic.

Does the problem with depth + color together still occur if you set enable_sync:=true in your launch instruction?

[draketmoore]

I have it enabled by default. When I turn set enable_sync to false and attempt to subscribe to both, the color stream is the one that stops publishing instead of the depth stream.

[MartyG-RealSense]

if sync can affect whether a stream works or not, what happens if the Global Time feature is disabled with depth_module.global_time_enabled:=false

[draketmoore]

I am only able to see depth_module.global_time_enabled parameter. Setting the value from the launch file or command line do not seem to change it. When I set the parameter manually after the realsense node has been launched, the issue still remains.

[MartyG-RealSense]

You are correct, it is depth_module.global_time_enabled:=false. I have edited my comment above - my apologies. :)

You listed at the top of this issue that you have a D405 and D435i. Is the color issue occurring with both cameras?

[draketmoore]

No worries! Yes, the issue persists with both of the cameras on my robot.

[MartyG-RealSense]

Are both cameras active simultaneously on the robot in a multi-camera streaming arrangement?

[draketmoore]

I will be running them simultaneously for my work, but at the moment during testing I am running them independently.

[MartyG-RealSense]

You could next test whether the order that streams are enabled makes a difference by first enabling depth only in the launch and then after launch, enable color during runtime.

ros2 param set /camera/camera enable_color true

[draketmoore]

Hi @MartyG-RealSense, sorry for the delay. When I launch with enable_color disabled, and monitor the depth image and then re-enable color, the depth stream stops alltogether. When I set it back to false, the depth resumes.

[MartyG-RealSense]

Reading this issue again from the beginning, I note that you have 5.15.0.2 listed as the firmware driver in your camera. That version was designed for librealsense 2.54.1. The recommended firmware for the 2.55.1 version of librealsense that you are using is 5.16.0.1.

[draketmoore]

Thanks, I just updated the firmware for both cameras to 5.16.0.1, but the issue remains.

[MartyG-RealSense]

On your wi-fi connection, are you using ssh to make the connection between the NUC on the robot and the desktop computer? If you are, does disabling the X11 forwarding function using the instructions at the link below make a difference?

https://www.simplified.guide/ssh/enable-x11-forwarding

[draketmoore]

I am not using ssh to connect at all, I am just running the realsense ros nodes directly on the NUC, and using a separate computer to subscribe to the topics.

[MartyG-RealSense]

The most difficult aspect of this case is that it is not clear why the depth stream would stop entirely on the NUC's side when the desktop subscribes to both depth and color if the NUC can publish both without problems before the desktop subscribes to them.

We also know that it is possible to subscribe to both depth and infrared without problems, but something about the color stream causes a failure with a 2-stream subscription. If depth + infrared works then it is not likely to be a problem with the desktop being over-burdened with processing both stream subscriptions.

I imagine that it might be too complicated to reverse the roles of the machines and try making the NUC the subscriber and the desktop the publisher?

Would it be possible for your project to use the infrared stream like a monochrome RGB image as a compromise if the color stream issue cannot be resolved?

[draketmoore]

Unfortunately, I am using the Stretch robot from Hello Robot, so there is no easy way to change which computer is using the realsense, but I will try with a separate realsense set up on the desktop and let you know if the issue persists.

Infrared as a substitute for color will not work for my specific use case, as I am using the desktop to run object detection models which require an RGB image.

[MartyG-RealSense]

Thanks very much. I look forward to your next report after trying a separate RealSense setup on the desktop. Good luck!

[draketmoore]

Hi @MartyG-RealSense, I just attached a separate D415 to my desktop, and I was successfully able to subscribe to both the color and depth topics. Not sure why this issue is only occurring in a single direction. Any advice/help would greatly appreciated. Thank you!

[MartyG-RealSense]

In cases where a RealSense camera is built into a non-Intel product such as a robot, we usually direct users to the support channel of the product's manufacturer, as Intel do not have the product in order to test it ourselves

Hello Robot provide their own ROS2 tutorial for using the RealSense camera with the Stretch robot.

https://docs.hello-robot.com/0.2/stretch-tutorials/ros2/deep_perception/

They have their own launch files, such as:

ros2 launch stretch_deep_perception stretch_detect_objects.launch.py

[MartyG-RealSense]

Hi @draketmoore Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.