search

intel-iot-devkit / intelligent-kiosk-analytics-cpp

Determine the demographics of an audience using the Intel® Distribution of OpenVINO™ toolkit, and then adjust the ads to match the audience.
https://software.intel.com/content/www/us/en/develop/topics/iot/reference-implementations/intelligent-digital-signage-solution.html
BSD 3-Clause "New" or "Revised" License
11 stars 12 forks source link
computer-vision deep-learning edge edge-ai edge-computing image-recognition inference intel live-demo machine-learning object-detection openvino pretrained-models real-time reference-implementation video

readme

DISCONTINUATION OF PROJECT

This project will no longer be maintained by Intel. Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project. Intel no longer accepts patches to this project. # 4K Advertisement Display on Digital Kiosk Based on Demographics

Details
Target OS: Ubuntu* 18.04 LTS
Programming Language: C++
Time to Complete: 45 min


Sample image

What it does

This reference implementation is developed using the Intel® Distribution of OpenVINO™ toolkit for audience analytics and Intel® Media SDK for the display of 4K advertisements. The application identifies the age and gender of the audience standing in front of digital signage, and based on the identification, it selects a suitable 4K advertisement. Real-time data visualization, which enables developers to monitor trends over time, occurs on Grafana*. This reference implementation aims to improve real-world marketing and advertising.

Requirements

Hardware

Software

How It Works

The application uses a video source, such as a camera to grab frames, and Deep Neural Networks (DNNs) to process data. The first network detects faces. The second neural network determines age and gender for each detected face. The third neural network detects the head pose of the person.

The application observes the age, gender, and head pose of the person standing in front of the digital signage camera and collects data for five frames. Data are normalized to avoid any fluctuations in the observation. Based on normalized data, the application selects a gender-appropriate advertisement.

A JSON file provides a list of advertisements for different age and gender groups. This JSON file is parsed, and data present in the file such as age group, gender, and a list of advertisements, are stored in a C++ structure. Once the software determines the dominant age-group and gender from the audience analytics, it selects an ad from JSON data, which is decoded using the HEVC plugin and played using Intel® Media SDK.

This application has two processes, the audience analytics process (AAP) and the advertisement display process (ADP). These processes use pipes for inter-process communication.

The AAP analyzes the audience, selects the appropriate ad, and writes the selected ad to the pipe, which the ADP reads from the other end of the pipe. The ADP reads the file name from the pipe, decodes the video which is in H.265 or hevc format, and renders it. Once the advertisement finishes playing, the ADP sends the acknowledgement of ad completion through the pipe to AAP. On receiving the acknowledgement, AAP selects the next ad based on the audience present at that time and sends it to ADP. The new advertisement plays only when the previous advertisement has completed.

The audience analysis happens even when an ad is playing as the two processes run in parallel. For every 20 frames, the application sends data to the InfluxDB database which is then visualized on Grafana to display the trends over time. Grafana visualizes the number of people who were interested in ads, the number of people not interested in ads, the ad currently playing, and the total number of unique visitors who were in front of the digital signage.

Code organization

Setup

Get the code

Clone the reference implementation

sudo apt-get update && sudo apt-get install git
git clone https://github.com/intel-iot-devkit/intelligent-kiosk-analytics-cpp.git

Install OpenVINO

Refer to Install the Intel® Distribution of OpenVINO™ toolkit for Linux* for more information about how to install and setup the Intel® Distribution of OpenVINO™ toolkit.

Install the OpenCL™ Runtime Package to run inference on the GPU, as shown in the instructions below. It is not mandatory for CPU inference.

Other dependencies

Intel® Media SDK
Intel® Media SDK provides a plain C API to access hardware-accelerated video decode, encode and filtering on Intel® Gen graphics hardware platforms.

**InfluxDB***
InfluxDB is a time series database designed to handle high write and query loads. InfluxDB is meant to be used as a backing store for any use case involving large amounts of timestamped data, including DevOps monitoring, application metrics, IoT sensor data, and real-time analytics.

**Grafana***
Grafana is an open-source, general purpose dashboard and graph composer, which runs as a web application. It supports Graphite, InfluxDB, Prometheus, OpenTSDB etc., as backends. Grafana allows you to query, visualize, alert on and understand your metrics no matter where they are stored.

Which models to use

This application uses the face-detection-retail-0004, age-gender-recognition-retail-0013 and head-pose-estimation-adas-0001 Intel® models, that can be downloaded using the model downloader. The model downloader downloads the .xml and .bin files that will be used by the application.

To download the models and install the dependencies of the application, run the below command in intelligent-kiosk-analytics-cpp directory:

./setup.sh

The config file

The resources/config.json contains the path of video that will be used by the application as input.

For example:

   {
       "inputs": [
          {
              "video":"path_to_video/video1.mp4",
          }
       ]
   }

The path/to/video is the path to an input video file.

Which Input Video to use

The application works with any input video. Sample videos are provided here.

For first-use, we recommend using face-demographics-walking-and-pause.

For example:

   {
       "inputs": [
          {
              "video":"sample-videos/face-demographics-walking-and-pause.mp4",
          }
       ]
   }

If the user wants to use any other video, it can be used by providing the path in the config.json file.

Using the Camera instead of video

Replace path/to/video with the camera ID in the config.json file, where the ID is taken from the video device (the number X in /dev/videoX).

On Ubuntu, to list all available video devices use the following command:

ls /dev/video*

For example, if the output of above command is /dev/video0, then config.json would be:

  {
     "inputs": [
        {
           "video":"0"
        }
     ]
  }

Setup the Environment

Go to the project directory:

cd <path-to-intelligent-kiosk-analytics-cpp>

Configure the environment to use the Intel® Distribution of OpenVINO™ toolkit by exporting environment variables:

source /opt/intel/openvino/bin/setupvars.sh

Build the Application

To build, go to intelligent-kiosk-analytics-cpp directory and run the following commands:

mkdir -p build && cd build
cmake ..
make

Run the Application

Run on the CPU

Although the application runs on the CPU by default, this can also be explicitly specified through the -d CPU _-dhp CPU _-dag CPU as command-line argument:

 __bin/Release/application -m  /opt/intel/openvino_2020.3.194/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-retail-0004/FP32/face-detection-retail-0004.xml -m_ag /opt/intel/openvino_2020.3.194/deployment_tools/open_model_zoo/tools/downloader/intel/age-gender-recognition-retail-0013/FP32/age-gender-recognition-retail-0013.xml -m_hp /opt/intel/openvino_2020.3.194/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP32/head-pose-estimation-adas-0001.xml

In addition to the model provided by the Intel® Distribution of OpenVINO™ toolkit, the user can include models trained on Caffe, TensorFlow, or Apache MXNet* machine learning frameworks for face-detection, age-gender recognition, or head pose. To do this, first configure the model optimizer for the framework used and then convert the trained model to produce an optimized Intermediate Representation (IR) of the model.

Learn more about the model optimizer in the Model Optimizer Developer Guide

Once the IR is obtained, run the application:

__bin/Release/application -m <path-to-face-detection-IR>/face-detection.xml -m_ag <path-to-age-gender-IR>/age-gender-recognition.xml -m_hp <path-to-head-pose-estimation-IR>/head-pose-estimation.xml

Note:

  1. To exit from the application, click on the Detection results window and long press the ESC key.
  2. To run the application on sync mode, use -async 0 as the command-line argument. By default the application runs on Async mode.
  3. To run with multiple devices use MULTI:device1,device2. For example: -d MULTI:CPU,GPU,MYRIAD

Running on different hardware

The application can use different hardware accelerator for different models. The user can specify the target device for each model using the command line argument as below:

-d <device>: Target device for Face Detection network (CPU, GPU, MYRIAD or HDDL).
-d_ag <device>: Target device for Age Gender Recognition network (CPU, GPU, MYRIAD or HDDL).
-d_hp <device>: Target device for Head Pose Estimation network (CPU, GPU, MYRIAD or HDDL).

For example: To run Face Detection model on CPU and Age Gender Recognition model with FP32 on GPU, Head Pose Estimation model on MYRIAD, use the below command:

__bin/Release/application -m /opt/intel/openvino_2020.3.194/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-retail-0004/FP32/face-detection-retail-0004.xml -m_ag /opt/intel/openvino_2020.3.194/deployment_tools/open_model_zoo/tools/downloader/intel/age-gender-recognition-retail-0013/FP32/age-gender-recognition-retail-0013.xml -m_hp /opt/intel/openvino_2020.3.194/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP32/head-pose-estimation-adas-0001.xml -d CPU -d_ag GPU -d_hp MYRIAD

FP32: FP32 is single-precision floating-point arithmetic uses 32 bits to represent numbers. 8 bits for the magnitude and 23 bits for the precision. For more information, click here
FP16: FP16 is half-precision floating-point arithmetic uses 16 bits. 5 bits for the magnitude and 10 bits for the precision. For more information, click here

Note: The Intel® Neural Compute Stick and HDDL can only run FP16 models. The model that is passed to the application, through the command-line argument, must be of data type FP16

Add New Advertisements to the List of Ads

To add the new ads in JSON file, follow this series of instructions:

  1. Video is first converted to H.265 or hevc format using FFmpeg* as the decoding process takes the input videos in this format for decoding. To convert an MP4 video to required format, follow these steps:

    • Install FFmpeg:

      sudo apt install ffmpeg

    • Convert video from .mp4 to .hevc format using FFmpeg:

      ffmpeg -i <input_video>.mp4 -c:v libx265 -c:a aac -b:a 128k <output_video>.hevc

  2. Copy the video to the resources directory found in the intelligent-kiosk-analytics-cpp directory.

  3. Add a video file name for an ad. For example, to add a new ad file called intel.hevc, which is meant to be played for males between the age group 28 to 50 years, modify the adlist.json file:

    • Open adList.json from intelligent-kiosk-analytics-cpp/resources directory in an editor.
    • Add the file name intel.hevc at the end of the list of the ads for "Age_Group": 3, "Gender": "M".
        {
        "Age_Group": 3,
        "Gender": "M",
        "Ads": [
            "maleAd.h265",
            "maleAd.h265",
            "maleAd.h265",
            "intel.hevc"
        ]
    }

    Note:

    • There are four age groups considered in the application:
      • Child: 1 to 13 yrs belongs to age group 1
      • Young Adult: 14 to 28 yrs belongs to age group 2
      • Adult: 28 to 50 yrs belongs to age group 3
      • Senior: 50+ yrs belongs to age group 4

InfluxDB Database

Visualize on Grafana