search

stereolabs / zed-yolo

3D Object detection using Yolo and the ZED in Python and C++
https://www.stereolabs.com/
MIT License
159 stars 68 forks source link

What is the latency and positioning accuracy in the ZED camera? #7

Closed AlexeyAB closed 2 years ago

AlexeyAB commented 5 years ago

Hi, I think about possibly the integration of some 3D camera support into https://github.com/AlexeyAB/darknet It's interesting that we can use ZED-camera on Jetson TX1/TX2/Xavier (L4T) to get 3D-coordinates (in meters) of detected objects relative to either Camera or World Frame (stationary point in the world):

I have a few questions:

adujardin commented 5 years ago

Hi Alexey,

The latency is the same for RGB and depth as the processing is synchronized by the grab() function. The latency depends on the load since the computation has to be finished before getting images and depth data. You can expect around 56ms on Xavier and 65ms (60 fps) on TX2 for RGB and depth with light settings, and around 190ms (24 fps) on TX2 for the highest (DEPTH_MODE_ULTRA, with depth stabilizer and tracking).

The length of the USB cable doesn't have an impact on the latency. However good quality, preferably shorter cables are more reliable since the bandwidth needed is very high. For long distances, an optic fiber can be used, you can check out our help center article for more info.

The positional error is highly dependent on many elements, mainly the type of motion (fast aggressive motion is harder to track) and the parameters. A higher framerate is always better to avoid motion blur and huge motions between frames.

For positional tracking error, our tests show in automotive scenario (e.g Kitti sequence like, with a quite low framerate), that the drift is around 2m at 100m. The loop closure can detect loop at 10m+ but is closer to ~1m jump on average in this configuration. On MAV sequence (EuRoC medium like, with good framerate and fast motions), the cumulative drift is around 1.2m at 80m.

AlexeyAB commented 5 years ago

Hi Aymeric,

Thanks! I answered your email. Also can we use the ZED camera together with the Oculus/Vive (mixed reality) to build a spatial map (SLAM), and then use the saved spatial map to move inside it only by using the Oculus/Vive without the ZED camera (virtual reality)? And what soft should we use for this?

adujardin commented 5 years ago

For mixed reality applications, we recommend the use of the ZED-Mini which have a baseline matching the eye and therefore can be comfortably used for stereo pass-through. We provide integration with Unity and Unreal 4 engines.

However, you can also build a textured mesh of the environment using the spatial mapping API available in the ZED SDK with a ZED or a ZED-Mini, it will output an .obj file. You can then load it as a standard mesh (containing geometry and textures) for VR visualization. This entire process can be done using Unity or Unreal alongside our plugins for the ZED SDK.

NB: You can also save the Positional tracking spatial memory, and then reload it using the ZED SDK. However, this is intended to match the same environment and it only affects the positional tracking performances and cannot be used for visualization.

pkr97 commented 5 years ago

Hi Alexey, Have you tried and implemented your idea of using ZED camera with darknet in Jetson module for object detection? How does it perform? How does it perform in case of static object detection and in case of dynamic object detection?

What is the accuracy during static object detection and tracking?

What is the accuracy in case of dynamic object detection and tracking?

I'm more interested in Jetson nano + ZED mini + darknet + OpenCV.

I shall be waiting for your valuable feedback.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days