Set up darknet zed ros

[chrstrom]

Object detection from darknet ros and depth estimation from the ZED camera should be fused in order to obtain a position estimate of the detected objects.

Subtasks

• [x] Get overveiw of ZED API, wrapper and darknet
• [x] Decide if darknet or ZED wrapper should be used for YOLO
• [x] Take a look at existing options for depth estimation
• [x] Record data for testing (subsea or over sea)
• [x] Run darknet zed to test object detection and depth estimation together on noctua pc
• [ ] Implement and run darknet zed with ros on noctua and xavier
• [ ] Figure out how to distribute the data recorded from darknet zed via ros
• [ ] Proof of concept
• [ ] Final implementation

[stefalarse]

After a quick search on stereolabs I found this page: https://www.stereolabs.com/docs/yolo/ and based on the first image it looks like our problem is solved (if we can run it of course...), or do we want something else/more detailed?

It does not seem to include ROS integration, so we might have to do that ourselves

[michoy]

Yes, that looks promising. Have you chekced if it is supported in the ROS wrapper?

[stefalarse]

Not sure how I check that:/

[michoy]

@Areskiko was looking at the ros wrapper earlier today, maybe he can help you out

[michoy]

Status update:

• Darknet ROS used outside of ZED for object detection
• ZED ROS Wrapper used as an interface to ZED measurements.
• Example script shall be used as inspiration for depth estimates. This script uses a point cloud from the ZED SDK (not ROS wrapper). This should be changed to getting the point cloud from a ROS topic, because it makes for a more modular system.

[theBadMusician]

We've successfully set up Darknet ROS ZED camera wrapper on Noctua and 'Hvit' PCs today. We didn't get the chance to test if the setup actually works with the camera, as it was inside of the Beluga camera casing today.

I've updated the Object Detection page in Vortex-AUV Wiki with the steps for setting up the system.

[mhiversflaten]

@theBadMusician An SVO-file from the last pool test can be found under Files/data/pool_test.svo in the Software channel. This includes footage from one of the objects of interest in regards to object detection.

[theBadMusician]

We've tested the ZED camera with the Darknet ROS ZED package on the Noctua PC.

Darknet ROS node publishes 3 topics: number of detected objects, bounding boxes' data (predicted objects' names, prediction certainties, depth/distance estimation, x/y - min/max coordinates of the bounding boxes), and the actual image view w/ object boxes. The data is per camera - if an object gets detected in the views of both left and right cameras, it registers as two separate objects.

I've tried setting up the package on the Xavier, but there were some compatibility issues with OpenCV, specifically OpenCV V4.x. This will most likely be solved by reflashing the Xavier (Issue #205).

What would be the next steps for this issue (besides setting up the package on Xavier)?

[michoy]

Great work! The next step would be to publish the depth image to the ros network. IMU and magnetometer readings would also be interesting to have on the network, as they might get useful in the future.

[chrstrom]

Taking all the work here and noting it down for the next team, but closing this issue so that we can start fresh for the new ppl :;)