Face detection (edge intelligence)

oberstet commented 6 years ago

We could expand the Alarm and/or the light detector app with the following:

1) detect faces on the camera shot taken 2) collect faces detected over the day on a summary page

Note: face detection is not yet fact recognition

The nice thing: this will be like 50 lines of Python, and we have a cluster ready (scale-out) face detection engine that is usable from any other WAMP component immediately.

Here are pointers:

This stuff is totally easy to use:

(cpy362_4) oberstet@thinkpad-t430s:~$ pip install opencv-python
Collecting opencv-python
  Downloading opencv_python-3.3.0.10-cp36-cp36m-manylinux1_x86_64.whl (15.4MB)
    100% |████████████████████████████████| 15.5MB 101kB/s 
Collecting numpy>=1.11.3 (from opencv-python)
  Downloading numpy-1.13.3-cp36-cp36m-manylinux1_x86_64.whl (17.0MB)
    100% |████████████████████████████████| 17.0MB 100kB/s 
Installing collected packages: numpy, opencv-python
Successfully installed numpy-1.13.3 opencv-python-3.3.0.10
(cpy362_4) oberstet@thinkpad-t430s:~$ cd scm/3rdparty/
(cpy362_4) oberstet@thinkpad-t430s:~/scm/3rdparty$ git clone https://github.com/shantnu/FaceDetect.git
Klone nach 'FaceDetect' ...
remote: Counting objects: 64, done.
remote: Total 64 (delta 0), reused 0 (delta 0), pack-reused 64
Entpacke Objekte: 100% (64/64), Fertig.
Prüfe Konnektivität ... Fertig.
(cpy362_4) oberstet@thinkpad-t430s:~/scm/3rdparty$ cd FaceDetect/
(cpy362_4) oberstet@thinkpad-t430s:~/scm/3rdparty/FaceDetect$ python face_detect.py abba.png
libpng warning: iCCP: profile 'Photoshop ICC profile': 'GRAY': Gray color space not permitted on RGB PNG
Traceback (most recent call last):
  File "face_detect.py", line 21, in <module>
    flags = cv2.cv.CV_HAAR_SCALE_IMAGE
AttributeError: module 'cv2.cv2' has no attribute 'cv'
(cpy362_4) oberstet@thinkpad-t430s:~/scm/3rdparty/FaceDetect$ python face_detect_cv3.py abba.png
libpng warning: iCCP: profile 'Photoshop ICC profile': 'GRAY': Gray color space not permitted on RGB PNG
Found 4 faces!
^C^CGetötet
(cpy362_4) oberstet@thinkpad-t430s:~/scm/3rdparty/FaceDetect$ subl face_detect_cv3.py
(cpy362_4) oberstet@thinkpad-t430s:~/scm/3rdparty/FaceDetect$ python face_detect_cv3.py abba.png^C
(cpy362_4) oberstet@thinkpad-t430s:~/scm/3rdparty/FaceDetect$ python face_detect_cv3.py abba.png
libpng warning: iCCP: profile 'Photoshop ICC profile': 'GRAY': Gray color space not permitted on RGB PNG
Found 4 faces!
^[[AGetötet
(cpy362_4) oberstet@thinkpad-t430s:~/scm/3rdparty/FaceDetect$ subl face_detect_cv3.py

oberstet commented 6 years ago

The component could have a super simpel WAMP API .. just 1 procedure:

io.crossbar.demo.cvengine.detect_faces(<PNG bytes>) -> [{x: 100, y: 121, w: 60, h: 56}, {x: 300, y: 290, w: 50, h: 48}]

It takes 1 positional argument (which is a binary string with the PNG image) and returns a list of detected face rectangles.

oberstet commented 6 years ago

The idea is to make this into a showcase for edge intelligence in particular: it does not make sense to upload raw images taken from a device camera into the cloud just to detect faces .. that can be done on the edge as well.

The component should be wrapped as a Docker container, and it should be possible to start multiple instances (by registering the procedure as "shared"). This allows us to scale out the face detector.

oberstet commented 6 years ago

OpenCV runs GPU accelerated:

https://opencv.org/platforms/opencl.html

This should be usable on devices such as

http://www.mouser.de/pdfdocs/UP-Square-DatasheetV05.pdf

which have Intel Gen 9 HD Graphics

om26er commented 6 years ago

I have the main stuff working (duh!), only need to pack that into a container.

import cv2
import numpy

async def get_faces_coordinates(image_data):
    # Create the haar cascade
    face_cascade = cv2.CascadeClassifier("cascades/haarcascade_frontalface_default.xml")

    image = numpy.fromstring(image_data, dtype=numpy.uint8)
    image_np = cv2.imdecode(image, cv2.IMREAD_ANYCOLOR)
    gray = cv2.cvtColor(image_np, cv2.COLOR_BGR2GRAY)

    # Detect faces in the image
    faces = face_cascade.detectMultiScale(gray, scaleFactor=1.2, minNeighbors=5, minSize=(30, 30))
    return [{'x': int(x), 'y': int(y), 'w': int(w), 'h': int(h)} for (x, y, w, h) in faces]

Tested above with a Python and a Java client. Works.

oberstet commented 6 years ago

@om26er awesome! yeah, containerization: so we need a Dockerfile that derives of the ABPy Docker image, adds the opencv/numpy stuff and adds/starts by default above image analyzer procedure, connecting to a CB instance with URL/realm/.. given via env var to Docker container.

With above containerized component, the image comes from a camera component somewhere. For the demo, we could have that camera component be a ABJS based Web thing that uses the Web camera API to get image frames, let them be analyzed using above image analyzer, and then draw over the detected rectangles over the image in the Web UI.

The backend image analyzer should register the procedure "shared", so we can start many instances of the Docker container. This is to demo the easy scaling.

om26er commented 6 years ago

I have actually working implementation here: https://github.com/om26er/wamp-face-detect which we will move to the iotcookbook.

The container there is based on ubuntu and packs crossbar with it. I will update it to remove crossbar from it and rebased on ABPy container.

oberstet commented 6 years ago

we should simplify that as much as possible, means, much less files, not like Java;) here: https://github.com/crossbario/crossbar-starter/tree/master/autobahn-python

actually, we should get away with 4 files: Makefile, app.py, requirements.txt, run

om26er commented 6 years ago

@oberstet I have now reduced the number of files, also rebased on autobahn-python:py3 docker container. PR here: https://github.com/crossbario/iotcookbook/pull/22

oberstet commented 6 years ago

@om26er

Great, so the PR is merged, and the face detection component is there in principle. However, it needs to be slightly modified / extended to demonstrates scale-out on machine clusters.

Here is what we want to demonstrate:

scaling up on a single box
scaling out over multiple boxes

For the first, assume a box is running on a 8 core CPU. We want to start 1 Docker container with the component on that box, but:

https://github.com/crossbario/iotcookbook/blob/master/device/edge/face-detect/app/server.py

=> get_faces_coordinates move to ApplicationSession (global functions WAMP registered are bad style), deferToThread onto a background threadpool of size 8 (3 lines or sth using Twisted machinery). => set max concurrency to 8 when registering the proc => register the proc shared

(https://github.com/crossbario/crossbar-examples/tree/master/scaling-microservices)

Then start 1 such component per box.

om26er commented 6 years ago

For the first, assume a box is running on a 8 core CPU

a somewhat implementation detail but is there a hard-requirement for 8-core or could we write code to just use a thread pool based on the number of available processor cores ?

oberstet commented 6 years ago

@om26er a command line option would be best: 0: autochoose (like you suggest based on number of cores), any other integer explicit threadpool size

om26er commented 6 years ago

scaling up on a single box

Implementation here: https://github.com/crossbario/iotcookbook/pull/24

oberstet commented 6 years ago

merged via https://github.com/crossbario/iotcookbook/pull/24

crossbario / iotcookbook

Face detection (edge intelligence) #20