search

stereolabs / zed-ros2-wrapper

ROS 2 wrapper for the ZED SDK
https://www.stereolabs.com/docs/ros2/
Apache License 2.0
134 stars 139 forks source link

Subscribing to zed2i images and running yolov7 reduces frame rate. #203

Closed yongjounwoo closed 3 months ago

yongjounwoo commented 3 months ago

Preliminary Checks

Description

First of all, my common.yaml and zed2i.yaml information is below. pub_frame_rate : 60.0 point_cloud_freq : 60.0 grab_frame_rate: 60

I subscribe to zed2i camera information through the zed_ros2_wrapper package and infer yolov7 in real time. At this time, when executing the yolov7 code, the frame rate of zed2i drastically decreases.

Steps to Reproduce

  1. zed_ros2_wrapper run(zed2i)
  2. yolov7_infer run
  3. frame rate reduce.. ...

Expected Result

I expected the frame to be maintained and the time delay to be 0.04 to 0.10 seconds, but that was not the case.

Actual Result

Before running yolov7 50~60 hz (time delay - 0.04~0.10)

After runnung yolov7 20~30 hz (time delay - 0.12 ~ 0.24)

ZED Camera model

ZED2i

Environment

GPU : Nvidia AGX Xavier
JetPack : 5.1.2 
ZED SDK : 4.0.8

I am using ZED2i.

Anything else?

No response

Myzhar commented 3 months ago

Hi @yongjounwoo this is not a bug of the wrapper. You are pushing the AGX Xavier to its limits, so a performance drop is expected. What depth mode are you using? Frame resolution?

Myzhar commented 3 months ago

I recommend posting on the StereoLabs forum for further discussions on this topic.

yongjounwoo commented 3 months ago

Hi! @Myzhar I'm using ULTRA depth mode and HD720.

Thank you so much for the quick reply.

Myzhar commented 3 months ago

I confirm that 60 FPS for such a resolution is optimistic. What's the inference time?

yongjounwoo commented 3 months ago

my inference time is 50~70fps. I used tensorrt.

also, I tried grab_frame_rate: 30 but the frame rate still dropped.

Myzhar commented 3 months ago

Have you tried to setup the DDS configuration to improve image transport performance?

yongjounwoo commented 3 months ago

I have not tried it yet, but I will give it a try

I observed that when I start the zed_ros2_wrapper and begin inference with YOLOv7, the frame rate of the ZED camera drops. However, if I stop the YOLOv7 inference, the frame rate of the ZED camera goes back up. Is the ZED SDK imposing a limitation?

Myzhar commented 3 months ago

The ZED SDK uses the GPU for depth and other processing. It's possible that the load of the inference engine is one of the cause.