darknet_ros_3d provides you 3d bounding boxes of the objects contained in an objects list, where is specificated the 3d position of each object. Using a RGBD camera like Asus Xtion, Orbbec Astra or Intel Realsense and neuronal network darknet ros, objects can be detected and his position can be calculated.
In addition, there is a visual debugger tool based on visual markers that you can see with rviz
The first we have to know is that darknet_ros_3d package have dependencies as follow:
You can install Darknet Ros following this steps. NOTE: ros2 branch
You must to clone gb_visual_detection_3d into src folder located in your workspace. Later, you have to compile it typing colcon build
.
First of all, is necessary to run camera driver. Make sure that camera driver is publishing point cloud information.
Later, you must run darknet_ros and, if everything worked properly, you should see 2d bounding boxes in your screen. If not, you have a problem with darknet_ros package.
Now, you can run darknet_ros_3d typing ros2 launch darknet_ros_3d darknet_ros_3d.launch.py
. If you want to change default parameters like topics it subscribe, you can change it in the configuration file located at ~/catkin_ws/src/gb_visual_detection_3d/darknet_ros_3d/darknet_ros_3d/config/
. Default parameters are the following:
interested_classes: Classes you want to detect. It must be classes names than exists previously in darknet ros.
mininum_detection_threshold: Maximum distance range between any pixel of image and the center pixel of the image to be considered.
minimum_probability: Minimum object probability (provided by darknet_ros) to be considered.
darknet_ros_topic: topic where darknet_ros publicates it's bounding boxes. /darknet_ros/bounding_boxes
.
point_cloud_topic: topic where point cloud is published from camera. By default: /camera/pointcloud
. It is important that point cloud topic be of PointCloud2 type and it be depth_registered.
working_frame: frame that all measurements are based on. By default, camera_link. It is very important that if you want to change this frame, it has the same axes than camera_link, if you would want 3d coordinates in another axis, you must change it later (once 3d bounding box has been calculated).
darknet3d_node provide bounding boxes. This bounding boxes are combinated with point cloud information to calculate (xmin, ymin, zmin) and (xmax, ymax, zmax) 3D coordinates.
Then, darknet_ros_3d publicates it's own bounding boxes array of BoundingBoxes3d type in /darknet_ros_3d/bounding_boxes_3d
by default, which is an array of BoundingBox3d that contains the following information:
std_msgs/Header header
BoundingBox3d[] bounding_boxes
BoundingBox3d:
string object_name
float64 probability
float64 xmin
float64 ymin
float64 xmax
float64 ymax
float64 zmin
float64 zmax
An example of output is as follow:
header:
stamp:
sec: 1593723845
nanosec: 430724839
frame_id: camera_link
bounding_boxes:
- object_name: person
probability: 0.7609682679176331
xmin: 0.4506256580352783
ymin: -0.3164764642715454
xmax: 0.7936256527900696
ymax: 0.11368180811405182
zmin: -0.25958430767059326
zmax: 0.10506562888622284
You can visualize the markers on rviz adding MarkerArray and subscribing to topic /darknet_ros_3d/markers
.