search

openvinotoolkit / open_model_zoo

Pre-trained Deep Learning models and demos (high quality and extremely fast)
https://docs.openvino.ai/latest/model_zoo.html
Apache License 2.0
4.1k stars 1.37k forks source link

OpenVino 2021.3 - Raspberry Pi 3B+ - #2406

Closed wonkydog closed 1 year ago

wonkydog commented 3 years ago

Hey,

Wondering if anyone would be so kind as to provide me with a little direction - I have spent a few weeks now trying to progress my understanding on OpenVino on the Rpi however seem to be missing something.

I have a very quick .py script working with the person-detection-retail-0013 model that is available within the open model zoo - However if I try and change the model (in this example a similar model such as person-detection-0203 I seem to run into problems. Running an image gives me the following error: "Inference Engine backend (device = MYRIAD): Reshape_75445 of type Reshape: Stage node Reshape_75445 (Reshape) types check error: output #0 has type FP16, but one of [S32] is expected in function 'initPlugin'"

I can't for the life of me work out what is wrong with the model or the blob? Do I need to optimise or convert models downloaded from the open model zoo? Would my blob need to be altered from the person-detection-retail-0013? apart from the input image size?

Models were downloaded from here: - name: FP16/person-detection-0203.xml size: 1236770 sha256: 33c4383fdcd5697819e6c1e3e4ed6ce341a85abf70ceb972d88b1eb991a4c760 source: https://storage.openvinotoolkit.org/repositories/open_model_zoo/2021.3/models_bin/2/person-detection-0203/FP16/person-detection-0203.xml - name: FP16/person-detection-0203.bin size: 3902628 sha256: 8beee3c9fe0a35efc17c4dd8f851caf69ecbb09a0578c6bf1631990857aeda8e source: https://storage.openvinotoolkit.org/repositories/open_model_zoo/2021.3/models_bin/2/person-detection-0203/FP16/person-detection-0203.bin

Example .py script (Working):


import json
import base64

import datetime
import os
import logging
from logging import info

from typing import List

import cv2 as cv
import numpy as np
from flask import (
    Flask,
    jsonify,
    request,
    send_file,
    send_from_directory,
    render_template,
)

app = Flask(__name__)

INPUT_FOLDER = "model/"  # where we read the neural network from

# the image size that the neural network is trained to work on.
# Should only be changed if the model is switched.
IMAGE_FORMAT = (544, 320)

# a pretrained model from OpenVino, see https://docs.openvinotoolkit.org/2018_R5/_docs_Retail_object_detection_pedestrian_rmnet_ssd_0013_caffe_desc_person_detection_retail_0013.html
NETWORK_NAME = "person-detection-retail-0013"
CONFIDENCE_THRESHOLD = 0.7  # when we make the cutoff on the bounding boxes

def recognize(img: np.ndarray, net) -> List:
    """
    recognizes and transforms an image
    """
    # Read an image.
    if img is None:
        info("no image to process")
        return []
    # Prepare input blob and perform inference.
    net.setInput(cv.dnn.blobFromImage(img, size=IMAGE_FORMAT, ddepth=cv.CV_8U))
    net_output = net.forward()
    print(net_output)

    rectangles = []
    # see https://docs.openvinotoolkit.org/2019_R3.1/_models_intel_face_detection_adas_0001_description_face_detection_adas_0001.html

    detections = net_output.reshape(-1, 7)
    for detection in detections:
        confidence = float(detection[2])
        if confidence >= CONFIDENCE_THRESHOLD:
            xmin = int(detection[3] * img.shape[1])
            ymin = int(detection[4] * img.shape[0])
            xmax = int(detection[5] * img.shape[1])
            ymax = int(detection[6] * img.shape[0])
            rectangles.append((xmin, ymin, xmax, ymax))
            info(f"detected #{len(detections)} people")

    return rectangles

NET = cv.dnn.readNet(
        os.path.join(INPUT_FOLDER, NETWORK_NAME + ".bin"),
        os.path.join(INPUT_FOLDER, NETWORK_NAME + ".xml"),
    )

NET.setPreferableTarget(cv.dnn.DNN_TARGET_MYRIAD)

@app.route("/get", methods=["GET"])
def main():

    return send_file("output.jpg")

@app.route("/post", methods=["POST"])
def post():
    image = request.get_json()
    image = image.get('image')
    image = base64.b64decode(image)

    image = np.frombuffer(image, np.uint8)
    image = cv.imdecode(image, cv.IMREAD_COLOR)
    rectangles = recognize(image, NET)
    print(rectangles)

    for rectangle in rectangles:
        cv.rectangle( image,(rectangle[0], rectangle[1]), (rectangle[2], rectangle[3]), color=(0, 255, 0), thickness=5)
        cv.imwrite('output.jpg', image)
    image = base64.b64encode(image)
    return send_file("output.jpg")

if __name__ == "__main__":
    app.run()

First time posting on git - I'm so sorry if this is a wrong category or place, more than happy to move to somewhere else if needed. Thanks for any help what so ever! More than happy to provide further info if required.

WD

vladimir-dudnik commented 3 years ago

@wonkydog do not worry, you posted at right place. Your issue might be related to the difference in output blobs format for those two models, please review person-detection-retail-0013 and person-detection-0203 descriptions. The first model has only single output blob, following standard SSD format [1, 1, N, 7], while second model have two output blobs, with format [N, 5] and [N] correspondingly.

I'd recommend you to try Open Model Zoo Object Detection Python demo, which supports both of these models.

vladimir-dudnik commented 3 years ago

@wonkydog have you solved your issue?

wonkydog commented 3 years ago

Yes, thank you so much! That’s really helped me out!

I have now been able to run a few different models providing they have the same input and output blob…

Blobs seem to be the bit I’m missing so I’m going to have a look into those.

Is there an easy way to filter or work the outputs where two blobs are given? Would something need to be set in net.forward() to select which blobs are to be returned?

Thanks again for any light you can shine! You’ve certainly got me going in the right direction now!

WD

wonkydog commented 3 years ago

Having another look this evening:

Still no luck with models other than those with one output! I must be doing something silly!

Anyone see the error of my ways?

Thanks all! Much appreciated!

WD


[setupvars.sh] OpenVINO environment initialized
pi@raspberrypi:~ $ cd /home/pi/Desktop
pi@raspberrypi:~/Desktop $ wget ttps://storage.openvinotoolkit.org/repositories/open_model_zoo/2021.3/models_bin/2/person-detection-0203/FP16/person-detection-0203.xml
ttps://storage.openvinotoolkit.org/repositories/open_model_zoo/2021.3/models_bin/2/person-detection-0203/FP16/person-detection-0203.xml: Unsupported scheme ‘ttps’.
pi@raspberrypi:~/Desktop $ wget https://storage.openvinotoolkit.org/repositories/open_model_zoo/2021.3/models_bin/2/person-detection-0203/FP16/person-detection-0203.xml
--2021-05-20 20:39:39--  https://storage.openvinotoolkit.org/repositories/open_model_zoo/2021.3/models_bin/2/person-detection-0203/FP16/person-detection-0203.xml
Resolving storage.openvinotoolkit.org (storage.openvinotoolkit.org)... 99.84.15.28, 99.84.15.70, 99.84.15.19, ...
Connecting to storage.openvinotoolkit.org (storage.openvinotoolkit.org)|99.84.15.28|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1236770 (1.2M) [application/xml]
Saving to: ‘person-detection-0203.xml’

person-detection-02 100%[===================>]   1.18M  5.72MB/s    in 0.2s    

2021-05-20 20:39:40 (5.72 MB/s) - ‘person-detection-0203.xml’ saved [1236770/1236770]

pi@raspberrypi:~/Desktop $ wget https://storage.openvinotoolkit.org/repositories/open_model_zoo/2021.3/models_bin/2/person-detection-0203/FP16/person-detection-0203.bin
--2021-05-20 20:39:54--  https://storage.openvinotoolkit.org/repositories/open_model_zoo/2021.3/models_bin/2/person-detection-0203/FP16/person-detection-0203.bin
Resolving storage.openvinotoolkit.org (storage.openvinotoolkit.org)... 99.84.15.60, 99.84.15.19, 99.84.15.70, ...
Connecting to storage.openvinotoolkit.org (storage.openvinotoolkit.org)|99.84.15.60|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3902628 (3.7M) [application/octet-stream]
Saving to: ‘person-detection-0203.bin’

person-detection-02 100%[===================>]   3.72M  7.17MB/s    in 0.5s    

2021-05-20 20:39:55 (7.17 MB/s) - ‘person-detection-0203.bin’ saved [3902628/3902628]

pi@raspberrypi:~/Desktop $ python3
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2 as cv
>>> image = cv.imread('motion11.jpg')
>>> image
array([[[ 35,  56,  88],
        [ 35,  56,  88],
        [ 35,  56,  88],
        ...,
        [ 94, 106, 134],
        [ 94, 106, 134],
        [ 94, 106, 134]],

       [[ 35,  56,  88],
        [ 35,  56,  88],
        [ 35,  56,  88],
        ...,
        [ 92, 106, 134],
        [ 94, 106, 134],
        [ 92, 106, 134]],

       [[ 35,  56,  88],
        [ 35,  56,  88],
        [ 35,  56,  88],
        ...,
        [ 90, 107, 133],
        [ 92, 107, 133],
        [ 90, 107, 133]],

       ...,

       [[ 28,  26,  32],
        [ 28,  26,  32],
        [ 28,  26,  32],
        ...,
        [ 88, 100, 124],
        [ 76,  88, 112],
        [ 57,  69,  93]],

       [[ 28,  26,  32],
        [ 28,  26,  32],
        [ 28,  26,  32],
        ...,
        [ 83,  95, 119],
        [ 74,  86, 110],
        [ 59,  71,  95]],

       [[ 28,  26,  32],
        [ 28,  26,  32],
        [ 28,  26,  32],
        ...,
        [ 76,  88, 112],
        [ 73,  85, 109],
        [ 66,  78, 102]]], dtype=uint8)
>>> blob = cv.dnn.blobFromImage(image, swapRB=True, size=(800,1280))
>>> blob
array([[[[ 88.,  88.,  88., ..., 134., 134., 134.],
         [ 88.,  88.,  88., ..., 134., 134., 134.],
         [ 88.,  88.,  88., ..., 133., 133., 133.],
         ...,
         [ 32.,  32.,  32., ..., 101., 118., 106.],
         [ 32.,  32.,  32., ..., 108., 116., 106.],
         [ 32.,  32.,  32., ..., 113., 114., 107.]],

        [[ 56.,  56.,  56., ..., 106., 106., 106.],
         [ 56.,  56.,  56., ..., 106., 106., 106.],
         [ 56.,  56.,  56., ..., 106., 106., 106.],
         ...,
         [ 26.,  26.,  26., ...,  77.,  94.,  82.],
         [ 26.,  26.,  26., ...,  84.,  92.,  82.],
         [ 26.,  26.,  26., ...,  89.,  90.,  83.]],

        [[ 35.,  35.,  35., ...,  94.,  94.,  94.],
         [ 35.,  35.,  35., ...,  93.,  94.,  94.],
         [ 35.,  35.,  35., ...,  92.,  92.,  92.],
         ...,
         [ 28.,  28.,  28., ...,  65.,  82.,  70.],
         [ 28.,  28.,  28., ...,  72.,  80.,  70.],
         [ 28.,  28.,  28., ...,  77.,  78.,  71.]]]], dtype=float32)
>>> blob.shape
(1, 3, 1280, 800)
>>> blob = cv.dnn.blobFromImage(image, swapRB=True, size=(864,480))
>>> blob.shape
(1, 3, 480, 864)
>>> net
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'net' is not defined
>>> net = cv.dnn.readNet('person-detection-0203.bin,'person-detection-0203.xml')
  File "<stdin>", line 1
    net = cv.dnn.readNet('person-detection-0203.bin,'person-detection-0203.xml')
                                                          ^
SyntaxError: invalid syntax
>>> net = cv.dnn.readNet('person-detection-0203.bin','person-detection-0203.xml')
>>> net
<dnn_Net 0x72a71730>
>>> net.setPreferableTarget(cv.dnn.DNN_TARGET_MYRIAD)
>>> net.setInput(blob)
>>> net.forward()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cv2.error: OpenCV(4.5.2-openvino) ../opencv/modules/dnn/src/ie_ngraph.cpp:718: error: (-2:Unspecified error) Failed to initialize Inference Engine backend (device = MYRIAD): Reshape_33569 of type Reshape: Stage node Reshape_33569 (Reshape) types check error: output #0 has type FP16, but one of [S32] is expected in function 'initPlugin'

>>> net.forward('boxes')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cv2.error: OpenCV(4.5.2-openvino) ../opencv/modules/dnn/src/ie_ngraph.cpp:718: error: (-2:Unspecified error) Failed to initialize Inference Engine backend (device = MYRIAD): Reshape_74083 of type Reshape: Stage node Reshape_74083 (Reshape) types check error: output #0 has type FP16, but one of [S32] is expected in function 'initPlugin'

>>> net.forward(['boxes'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cv2.error: OpenCV(4.5.2-openvino) ../opencv/modules/dnn/src/ie_ngraph.cpp:718: error: (-2:Unspecified error) Failed to initialize Inference Engine backend (device = MYRIAD): Reshape_114597 of type Reshape: Stage node Reshape_114597 (Reshape) types check error: output #0 has type FP16, but one of [S32] is expected in function 'initPlugin'
vladimir-dudnik commented 3 years ago

@wonkydog OpenVINO OMZ samples should also work on RasperryPi, they are based on OpenVINO API, OpenCV DNN API might have some limitations @allnes @mshabunin do you have an OpenCV DNN sample which showcase OMZ ‘person-detection-0203 model inference?

mshabunin commented 3 years ago

OpenCV passes models in OpenVINO IR format directly to OpenVINO backend. It seems that this model is not supported on MYRIAD. I tried to run object_detection_demo on x86_64 Ubuntu 18 machine (both FP16 and FP32 models) and it prints the same result:

$ ./object_detection_demo -m person-detection-0203.xml -i ~/Videos/sintel_trailer-720p.mp4 -no_show -at ssd -d MYRIAD
[ INFO ] InferenceEngine:       API version ......... 2.1
        Build ........... 2021.4.0-3532-6bb5a2c3cb1
[ INFO ] Parsing input parameters
[ INFO ] Reading input
[ INFO ] Loading Inference Engine
[ INFO ] Device info:
[ INFO ]        MYRIAD
        myriadPlugin version ......... 2.1
        Build ........... 2021.4.0-3532-6bb5a2c3cb1
Loading network files
[ INFO ] Batch size is forced to 1.
[ INFO ] Checking that the inputs are as the demo expects
[ INFO ] Checking that the outputs are as the demo expects
[ INFO ] Loading model to the device
[ ERROR ] Reshape_19957 of type Reshape: [ GENERAL_ERROR ]
/home/jenkins/agent/workspace/private-ci/ie/build-linux-ubuntu18/b/repos/openvino/inference-engine/src/vpu/graph_transformer/src/model/stage.cpp:340 Stage node Reshape_19957 (Reshape) types check error: output #0 has type FP16, but one of [S32] is expected
vladimir-dudnik commented 2 years ago

@wonkydog seems this was answered, do you have other questions?