ZED is very slow

[ArtlyStyles]

Hi,

I am running ZED ROS wrapper on MIT-Racecar, a car-like robot. The node is started by zed_wrapper/zed.launch. When the cars was driving, I monitored left camera rectified image through rviz. However, the image was not continous, I got about 1 frame update every 5 seconds. The car has jetson Tx2 so computation should not be a problem.

Actually for my application, I only need the raw images from both cameras to feed a deep learning neural network. I don't need deep image, rectified image, etc.

How can I improve the frame rate?

[Myzhar]

Have you verified that the Jetson TX2 is in "performance" mode and not in "on demand"? Try to execute sudo ~/jetsonclocks.sh before starting the ZED node to be sure that the maximum computational power of the TX2 is used. Also check the camera resolution, the Jetson TX2 can support up to 60 FPS at 720p

[ArtlyStyles]

Hi @Myzhar , thank you for the suggestion. Could you tell me how to check and set camera resolution?

[Myzhar]

Check the file "zed_camera.launch" you will find two parameters "resolution" and "framerate" and the possible setting combinations

[ArtlyStyles]

In the launch file, gpu_id=-1, should I change that to gpu_id=0? Since I do have GPU on my TX2

[Myzhar]

gpu=-1 means that the first available GPU will be used. In your case it is like setting it to 0.

[ArtlyStyles]

Thanks. After changing the resolution and frame rate, the image is much smoother. However, I found there was still some delay. Since I am going to use the image to train a deep learning network and drive the car, the image need to be paired with control parameters. So the delay is a big problem for me. Is there a way to fix it? Also, since for this application I do not need the depth image, can I turn 3D reconstruct /deep sense off? I want use ZED as just a now camera.

[Myzhar]

The wrapper is "smart": https://github.com/stereolabs/zed-ros-wrapper/blob/master/src/nodelet/src/zed_wrapper_nodelet.cpp#L1264 If there are no subscribers to "depth" topics the depth is not calculated.

About latency, it is a "ROS issue" since the ZED SDK introduces a very small latency. To reduce ROS latency I suggest you to subscribe to image topics using UDP instead of TCP (see 'transport_hints') or to use the "nodlets intraprocess communication", since ZED wrapper can be used as a nodelet loaded by an external nodlets manager

[ArtlyStyles]

I am using rosbag to save the images. Can I use UDP in rosbag?

[Myzhar]

https://github.com/ros/ros_comm/issues/1279

[ArtlyStyles]

Hi not sure what #1279 means. Is that already available in rosbag?

[Myzhar]

@ArtlyStyles this is a ROS related issue. You can find more information on ROS channels

[ArtlyStyles]

Hi @Myzhar I would like to study more on the latency issue. Can I view ZED image topic in rviz using UDP?

[Myzhar]

Sure you can 1) add an Image viewer 2) choose the image topic from the list of available topic 3) choose "raw" as transport, otherwise you will add compression process to latency 4) check "unreliable" to switch from TCP to UDP

Please remember that the latency is due to ROS communication and Rviz processing and what you see on Rviz is not the real ZED latency that can be measured at max as 3 or 4 frames on Ubuntu.

[ArtlyStyles]

Hi @Myzhar Thanks. I tried that, the improvement was not very obvious. Since this is a ROS issue, it probably means that ROS is not a good choice for time-critical projects, such as autonomous driving? And changing to another camera won't help?

[Myzhar]

This is one of the critics made to ROS. ROS2 is under heavy development and it has a full new communication structure. The first version has been release (Bouncy), but it's not yet ready for a final product

[ArtlyStyles]

I am thinking writing my own ROS node which talks to ZED API directly and output driving commands. If I copy zed_wrapper_nodelet.cpp, modify it by putting my image processing code in and then output 2 floating point numbers through ROS publisher, then there should not be any delay. Correct?

[Myzhar]

Not really correct, you will have a delay generated by the sum of two factors: 1) the ZED delay, that is about 2-3 frames under Linux 2) the time you take for you image processing 3) the time the 2 floating point numbers propagate through ROS network

[ArtlyStyles]

yes, I will have these delays. But it should be much shorter since it will take about half second just to get the image from ROS

[Myzhar]

I close the issue, but you can continue to post comments if you need further help

[CrossWax]

Hello @Myzhar, I have been following this topic closeley, and have found my self in a similar situation. I subscribe to the ZED 1, through the left/rect_image_color topic. On ZEDs side, everything seems grate. Constante 15Hz publishing rate. I started by using the topic delay to measure the elapsed time of the message, and the ros tool mentions its somewhere around 100ms ( when you are publishing every 66ms). In any case, I then measure the node that is subscribing, and I get that the publishing rate has droped to 3.7 Hz. When I measure it using rclcpp system clock instance, and set it just after it subscribes, I get the same instance, with a rate of 3.7 Hz.

Also I am using ROS 2 Foxy, on a Nvidia AGX Orion. I thought it would be something I could solve leveraging cuda or tapping into the more powerfull board, but nothing so far.

Any suggestions on how could I proceed? Thank you

[Myzhar]

@CrossWax this is a closed issue. Please do not reply to closed issues. It's a good practice to open a new issue, add all the required information and eventually cite a closed issue if eventually required. Furthermore, your problem is with ROS 2 and this is the ROS 1 repository.

[CrossWax]

Appologies @Myzhar , I have now created a new issue https://community.stereolabs.com/t/latency-in-using-zed-with-ros-1-and-ros-2/3499. Thank you