This project will no longer be maintained by Intel.
Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.
Intel no longer accepts patches to this project.
If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the open source software community, please create your own fork of this project.
中文文档入口: https://github.com/intel/ros_openvino_toolkit/tree/dev-ov2021.4/doc/ZH-CN
The OpenVINO™ (Open visual inference and neural network optimization) toolkit provides a ROS-adaptered runtime framework of neural network which quickly deploys applications and solutions for vision inference. By leveraging Intel® OpenVINO™ toolkit and corresponding libraries, this runtime framework extends workloads across Intel® hardware (including accelerators) and maximizes performance.
From the view of hirarchical architecture design, the package is divided into different functional components, as shown in below picture.
From the view of logic implementation, the package introduces the definitions of parameter manager, pipeline and pipeline manager. The below picture depicts how these entities co-work together when the corresponding program is launched.
Once a corresponding program is launched with a specified .yaml config file passed in the .launch file or via commandline, parameter manager analyzes the configurations about pipeline and the whole framework, then shares the parsed configuration information with pipeline procedure. A pipeline instance is created by following the configuration info and is added into pipeline manager for lifecycle control and inference action triggering.
The contents in .yaml config file should be well structured and follow the supported rules and entity names. Please see the configuration guidance for how to create or edit the config files.
Pipeline fulfills the whole data handling process: initiliazing Input Component for image data gathering and formating; building up the structured inference network and passing the formatted data through the inference network; transfering the inference results and handling output, etc.
Pipeline manager manages all the created pipelines according to the inference requests or external demands (say, system exception, resource limitation, or end user's operation). Because of co-working with resource management and being aware of the whole framework, it covers the ability of performance optimization by sharing system resource between pipelines and reducing the burden of data copy.
Currently, the package support several kinds of input resources of gaining image data:
Input Resource | Description |
---|---|
StandardCamera | Any RGB camera with USB port supporting. Currently only the first USB camera if many are connected. |
RealSenseCamera | Intel RealSense RGB-D Camera, directly calling RealSense Camera via librealsense plugin of openCV. |
Image Topic | any ROS topic which is structured in image message. |
Image File | Any image file which can be parsed by openCV, such as .png, .jpeg. |
Video File | Any video file which can be parsed by openCV. |
Currently, the inference feature list is supported:
Inference | Description |
---|---|
Face Detection | Object Detection task applied to face recognition using a sequence of neural networks. |
Emotion Recognition | Emotion recognition based on detected face image. |
Age & Gender Recognition | Age and gener recognition based on detected face image. |
Head Pose Estimation | Head pose estimation based on detected face image. |
Object Detection | object detection based on SSD-based trained models. |
Vehicle Detection | Vehicle and passenger detection based on Intel models. |
Object Segmentation | object detection and segmentation. |
Person Reidentification | Person Reidentification based on object detection. |
/camera/color/image_raw
(sensor_msgs::Image)
/ros_openvino_toolkit/face_detection
(object_msgs::ObjectsInBoxes)/ros_openvino_toolkit/emotions_recognition
(vino_people_msgs::EmotionsStamped)
Age and Gender Recognition:
/ros_openvino_toolkit/age_genders_Recognition
(vino_people_msgs::AgeGenderStamped)
Head Pose Estimation:
/ros_openvino_toolkit/headposes_estimation
(vino_people_msgs::HeadPoseStamped)
Object Detection:
/ros_openvino_toolkit/detected_objects
(object_msgs::ObjectsInBoxes)
Object Segmentation:
/ros_openvino_toolkit/segmented_obejcts
(vino_people_msgs::ObjectsInMasks)
Person Reidentification:
/ros_openvino_toolkit/reidentified_persons
(vino_people_msgs::ReidentificationStamped)
Rviz Output:
/ros_openvino_toolkit/image_rviz
(sensor_msgs::Image)
/detect_object
(object_msgs::DetectObject)/detect_face
(object_msgs::DetectObject)/detect_age_gender
(vino_people_msgs::AgeGender)/detect_head_pose
(vino_people_msgs::HeadPose)/detect_emotion
(vino_people_msgs::Emotion)RViz dispaly is also supported by the composited topic of original image frame with inference result.
To show in RViz tool, add an image marker with the composited topic:
/ros_openvino_toolkit/image_rviz
(sensor_msgs::Image)
OpenCV based image window is natively supported by the package. To enable window, Image Window output should be added into the output choices in .yaml config file. see the config file guidance for checking/adding this feature in your launching.
See below pictures for the demo result snapshots.
face detection input from standard camera
object detection input from realsense camera
object segmentation input from video
Person Reidentification input from standard camera
NOTE: Intel releases 2 different series of OpenVINO Toolkit, we call them as OpenSource Version and Tarball Version. This guidelie uses OpenSource Version as the installation and launching example. If you want to use Tarball version, please follow the guide for Tarball Version.
One-step installation scripts are provided for the dependencies' installation. Please see the guide for details.
Preparation
cd ~/Downloads
cat <<EOF > 97-usbboot.rules
SUBSYSTEM=="usb", ATTRS{idProduct}=="2150", ATTRS{idVendor}=="03e7", GROUP="users", MODE="0666", ENV{ID_MM_DEVICE_IGNORE}="1"
SUBSYSTEM=="usb", ATTRS{idProduct}=="2485", ATTRS{idVendor}=="03e7", GROUP="users", MODE="0666", ENV{ID_MM_DEVICE_IGNORE}="1"
SUBSYSTEM=="usb", ATTRS{idProduct}=="f63b", ATTRS{idVendor}=="03e7", GROUP="users", MODE="0666", ENV{ID_MM_DEVICE_IGNORE}="1"
EOF
sudo cp 97-usbboot.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules
sudo udevadm trigger
sudo ldconfig
rm 97-usbboot.rules
#object segmentation model
cd /opt/openvino_toolkit/dldt/model-optimizer/install_prerequisites
sudo ./install_prerequisites.sh
mkdir -p ~/Downloads/models
cd ~/Downloads/models
wget http://download.tensorflow.org/models/object_detection/mask_rcnn_inception_v2_coco_2018_01_28.tar.gz
tar -zxvf mask_rcnn_inception_v2_coco_2018_01_28.tar.gz
cd mask_rcnn_inception_v2_coco_2018_01_28
#FP32
sudo python3 /opt/openvino_toolkit/dldt/model-optimizer/mo_tf.py --input_model frozen_inference_graph.pb --tensorflow_use_custom_operations_config /opt/openvino_toolkit/dldt/model-optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config pipeline.config --reverse_input_channels --output_dir /opt/openvino_toolkit/models/segmentation/output/FP32
#FP16
sudo python3 /opt/openvino_toolkit/dldt/model-optimizer/mo_tf.py --input_model frozen_inference_graph.pb --tensorflow_use_custom_operations_config /opt/openvino_toolkit/dldt/model-optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config pipeline.config --reverse_input_channels --data_type=FP16 --output_dir /opt/openvino_toolkit/models/segmentation/output/FP16
cd /opt/openvino_toolkit/open_model_zoo/tools/downloader
sudo python3 downloader.py --name face-detection-adas-0001 --output_dir /opt/openvino_toolkit/models/face_detection/output
sudo python3 downloader.py --name age-gender-recognition-retail-0013 --output_dir /opt/openvino_toolkit/models/age-gender-recognition/output
sudo python3 downloader.py --name emotions-recognition-retail-0003 --output_dir /opt/openvino_toolkit/models/emotions-recognition/output
sudo python3 downloader.py --name head-pose-estimation-adas-0001 --output_dir /opt/openvino_toolkit/models/head-pose-estimation/output
sudo python3 downloader.py --name person-detection-retail-0013 --output_dir /opt/openvino_toolkit/models/person-detection/output
sudo python3 downloader.py --name person-reidentification-retail-0076 --output_dir /opt/openvino_toolkit/models/person-reidentification/output
sudo python3 downloader.py --name vehicle-license-plate-detection-barrier-0106 --output_dir /opt/openvino_toolkit/models/vehicle-license-plate-detection/output
sudo python3 downloader.py --name vehicle-attributes-recognition-barrier-0039 --output_dir /opt/openvino_toolkit/models/vehicle-attributes-recongnition/output
sudo python3 downloader.py --name license-plate-recognition-barrier-0001 --output_dir /opt/openvino_toolkit/models/license-plate-recognition/output
sudo python3 downloader.py --name landmarks-regression-retail-0009 --output_dir /opt/openvino_toolkit/models/landmarks-regression/output
sudo python3 downloader.py --name face-reidentification-retail-0095 --output_dir /opt/openvino_toolkit/models/face-reidentification/output
copy label files (excute once)
sudo cp /opt/openvino_toolkit/ros_openvino_toolkit/data/labels/emotions-recognition/FP32/emotions-recognition-retail-0003.labels /opt/openvino_toolkit/models/emotions-recognition/output/intel/emotions-recognition-retail-0003/FP32/
sudo cp /opt/openvino_toolkit/ros_openvino_toolkit/data/labels/face_detection/face-detection-adas-0001.labels /opt/openvino_toolkit/models/face_detection/output/intel/face-detection-adas-0001/FP32/
sudo cp /opt/openvino_toolkit/ros_openvino_toolkit/data/labels/face_detection/face-detection-adas-0001.labels /opt/openvino_toolkit/models/face_detection/output/intel/face-detection-adas-0001/FP16/
sudo cp /opt/openvino_toolkit/ros_openvino_toolkit/data/labels/object_segmentation/frozen_inference_graph.labels /opt/openvino_toolkit/models/segmentation/output/FP32/
sudo cp /opt/openvino_toolkit/ros_openvino_toolkit/data/labels/object_segmentation/frozen_inference_graph.labels /opt/openvino_toolkit/models/segmentation/output/FP16/
sudo cp /opt/openvino_toolkit/ros_openvino_toolkit/data/labels/object_detection/vehicle-license-plate-detection-barrier-0106.labels /opt/openvino_toolkit/models/vehicle-license-plate-detection/output/intel/vehicle-license-plate-detection-barrier-0106/FP32
sudo cp /opt/openvino_toolkit/ros_openvino_toolkit/data/labels/face_detection/face-detection-adas-0001.labels /opt/openvino_toolkit/models/face_detection/output/intel/face-detection-adas-0001/FP32/
sudo cp /opt/openvino_toolkit/ros_openvino_toolkit/data/labels/face_detection/face-detection-adas-0001.labels /opt/openvino_toolkit/models/face_detection/output/intel/face-detection-adas-0001/FP16/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openvino_toolkit/dldt/inference-engine/bin/intel64/Release/lib
roslaunch vino_launch pipeline_people.launch
roslaunch vino_launch pipeline_image.launch
roslaunch vino_launch pipeline_segmentation.launch
roslaunch vino_launch pipeline_video.launch
roslaunch vino_launch pipeline_reidentification.launch
roslaunch vino_launch pipeline_face_reidentification.launch
roslaunch vino_launch pipeline_vehicle_detection.launch
roslaunch vino_launch image_object_server.launch
Run example application with an absolute path of an image on another console:
rosrun vino_sample image_object_client ~/catkin_ws/src/ros_openvino_toolkit/data/images/car.png
roslaunch vino_launch image_people_server.launch
Run example application with an absolute path of an image on another console:
rosrun vino_sample image_people_client ~/catkin_ws/src/ros_openvino_toolkit/data/images/team.jpg