search

openvinotoolkit / openvino

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference
https://docs.openvino.ai
Apache License 2.0
7.34k stars 2.29k forks source link

Crash in net.forward() using inference engine in OpenCV on Raspberry Pi 4 and Intel NCS2 #1477

Closed ark- closed 4 years ago

ark- commented 4 years ago

Versions

Background

I am getting issues running a Darknet Yolov3-Tiny model with the OpenCV inference engine enhanced by the Intel Neural Compute Stick 2. I use OpenCV in CPU only mode it runs fine (but of course slow as it is not accelerated) which is why I'm posting in the OpenVino repo.

The same application code and model ran fine on the following combination of versions:

The reason I want to upgrade is to leverage some of the Yolo optimisations in OpenCV 4.4.

Issue

My application is written in Python 3.7 and when calling the net.forward() I get the following error:

terminate called after throwing an instance of 'ngraph::NodeValidationFailure'
  what():  Check 'is_valid_permutation(permutation, arg_shape.rank())' failed at /home/pi/openvino/ngraph/src/ngraph/op/transpose.cpp:65:
While validating node 'v1::Transpose Transpose_202(Add_200[0]:f32{1,72,13,13}, Constant_201[0]:i64{4}) -> (dynamic?)':
Permutation AxisVector{0, 3, 336, 0} is not valid for input shape {1,72,13,13}

or

terminate called after throwing an instance of 'ngraph::NodeValidationFailure'
  what():  Check 'is_valid_permutation(permutation, arg_shape.rank())' failed at /home/pi/openvino/ngraph/src/ngraph/op/transpose.cpp:65:
While validating node 'v1::Transpose Transpose_202(Add_200[0]:f32{1,72,13,13}, Constant_201[0]:i64{4}) -> (dynamic?)':
Permutation AxisVector{0, 3, 104, 1} is not valid for input shape {1,72,13,13}

This looks like theres a layer that's not been implemented in ngraph but whats confusing is that this worked fine in Openvino 2020.3.

Reproduce

This is being built on target. i.e. on the Raspberry Pi 4.

  1. Checkout versions of repos from first section of this issue. Assume both opencv and openvino are in $HOME
  2. apt-get all prerequisites
  3. Increase swap partition to 2048
  4. Build OpenVino with following cmake config:
    -D CMAKE_BUILD_TYPE=Release
    -D ENABLE_SSE42=OFF
    -D THREADING=SEQ
    -D ENABLE_GNA=OFF
    -D ENABLE_OPENCV=OFF
    -D ENABLE_PYTHON=ON
    -D CMAKE_CXX_FLAGS="-march=armv7-a -latomic"
    -D PYTHON_EXECUTABLE=/usr/bin/python3
    -D PYTHON_LIBRARY=/usr/lib/arm-linux-gnueabihf/libpython3.7m.so
    -D ENABLE_MKL_DNN=OFF
    -D ENABLE_CLDNN=OFF
    -D PYTHON_INCLUDE_DIR=/usr/include/python3.7m
  1. "Install" openvino by setting paths (this is so OpenCV can find the inference engine when building)
export LD_LIBRARY_PATH=/usr/local/lib:$HOME/openvino/bin/armv7l/Release/lib
export INTEL_OPENVINO_DIR=$HOME/openvino
export INTEL_CVSDK_DIR=$HOME/openvino
export InferenceEngine_DIR=$HOME/openvino/build
export ngraph_DIR=$HOME/openvino/build/ngraph
  1. Build OpenCV (with contrib modules) using the following cmake config:
    # Default from https://github.com/opencv/opencv/wiki/Intel's-Deep-Learning-Inference-Engine-backend
    -D CMAKE_BUILD_TYPE=RELEASE
    -D CMAKE_INSTALL_PREFIX=/usr/local
    -D ENABLE_NEON=ON
    -D CPU_BASELINE="NEON"
    -D ENABLE_VFPV3=ON
    -D ENABLE_CXX11=ON
    # Match flags from Openvino
    -D CMAKE_EXE_LINKER_FLAGS="-latomic"
    -D CMAKE_CXX_FLAGS="-march=armv7-a -latomic"
    # Disable unused examples and tests
    -D BUILD_EXAMPLES=OFF
    -D INSTALL_PYTHON_EXAMPLES=OFF
    -D BUILD_PERF_TESTS=OFF
    -D BUILD_TESTS=OFF
    -D WITH_IPP=OFF
    # Additional Modules
    -D OPENCV_EXTRA_MODULES_PATH="$HOME/opencv/opencv_contrib/modules"
    -D OPENCV_ENABLE_NONFREE=ON
    # Python Paths
    -D PYTHON2_INCLUDE_PATH=/usr/include/python2.7
    -D PYTHON2_NUMPY_INCLUDE_DIRS="$HOME/.local/lib/python2.7/site-packages/numpy/core/include"
    -D PYTHON3_INCLUDE_PATH=/usr/include/python3.7m
    -D PYTHON3_NUMPY_INCLUDE_DIRS="$HOME/.local/lib/python3.7/dist-packages/numpy/core/include"
    -D PYTHON3_CVPY_SUFFIX=.cpython-37m-arm-linux-gnueabihf.so
    -D BUILD_opencv_python3=yes
    # Inference Engine Paths
    -D WITH_INF_ENGINE=ON
    -D INF_ENGINE_RELEASE="2020040000"
    -D INF_ENGINE_LIB_DIRS="$HOME/openvino/bin/armv7l/Release/lib"
    -D INF_ENGINE_INCLUDE_DIRS="$HOME/openvino/inference-engine/include"
    -D CMAKE_FIND_ROOT_PATH="$HOME/openvino/"
    -D ENABLE_PRECOMPILED_HEADERS=OFF # https://github.com/opencv/opencv/issues/15278#issuecomment-522219507
  1. Make, make install and ldconfig
  2. Set up USB rules: https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_raspbian.html

The python script simplified is 

dnn_net = cv2.dnn.readNet(darknet_cfg_file, darknet_weights_file)
dnn_net.setPreferableBackend(cv2.dnn.DNN_BACKEND_INFERENCE_ENGINE)       
dnn_net.setPreferableTarget(cv2.dnn.DNN_TARGET_MYRIAD)

# As per OpenCV example defaults
blob = cv2.dnn.blobFromImage( cv2.resize(frame, (416, 416)),
            1 / 255, (416, 416), (10, 20, 30), swapRB=True, crop=False)

self.dnn_net.setInput(blob)

out = self.dnn_net.forward()
# Crash here
avitial commented 4 years ago

Hi @ark- thanks for reaching out. May I ask if this issue occurs with any of the available pre-trained models in OMZ? Or does this only occur with your YOLO implementation? Please help me provide the MO command used to convert to IR and if possible also the actual model files (.pb, .cfg/.weights files) if possible.

Instead of using OpenCV dnn module, by any chance have you tried using OpenVINO inference engine through IECore? Just curious if you see the same error.

Regards, Luis

ark- commented 4 years ago

May I ask if this issue occurs with any of the available pre-trained models in OMZ?

Unfortunately I have not tried. Allow me to investigate.

Or does this only occur with your YOLO implementation? Please help me provide the MO command used to convert to IR and if possible also the actual model files (.pb, .cfg/.weights files) if possible.

I did not convert it. I used as-is using the OpenCV readNetFromDarknet function DOC LINK. This method worked fine on OpenCV4.3 and Openvino 2020.3. I could load and run my models.

Instead of using OpenCV dnn module, by any chance have you tried using OpenVINO inference engine through IECore? Just curious if you see the same error.

I haven't used IECore. Allow me to read some docs and see what I can do.

ark- commented 4 years ago

Ok so I have tried some models from the Model Zoo here (https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin). 

dnn_net = cv2.dnn.readNet("person-detection-0106.bin", "person-detection-0106.xml", "dldt")
dnn_out_names = self.dnn_net.getUnconnectedOutLayersNames()

blob = cv2.dnn.blobFromImage(frame,
            1 / 255, (416, 416), (10, 20, 30),
            swapRB=True, crop=False)

dnn_net.setInput(blob)

output_blobs = self.dnn_net.forward(self.dnn_out_names)

Models tested are:

Results as follows:

Error for Person Detection 0106 (FP16):

terminate called after throwing an instance of 'InferenceEngine::details::InferenceEngineException'
  what():  Check 'PartialShape::broadcast_merge_into(pshape, get_input_partial_shape(i), autob)' failed at /home/pi/openvino/ngraph/src/ngraph/node.cpp:922:
While validating node 'v1::Add Add_3000(Add_2964[0]:f16{1,256,26,26}, Interpolate_2999[0]:f16{1,256,50,84}) -> (f16{1,256,50,84})':
Argument shapes are inconsistent.

Error for Person Detection 0106 (FP32):

terminate called after throwing an instance of 'InferenceEngine::details::InferenceEngineException'
  what():  Check 'PartialShape::broadcast_merge_into(pshape, get_input_partial_shape(i), autob)' failed at /home/pi/openvino/ngraph/src/ngraph/node.cpp:922:
While validating node 'v1::Add Add_3000(Add_2964[0]:f32{1,256,26,26}, Interpolate_2999[0]:f32{1,256,50,84}) -> (f32{1,256,50,84})':
Argument shapes are inconsistent.

Pedestrian Detection ADAS 0002 (FP32) didn't crash but I am only receiving really low confidences. I switched the input video size to match those listed here: https://docs.openvinotoolkit.org/2019_R1/_pedestrian_detection_adas_0002_description_pedestrian_detection_adas_0002.html

Person Detection Retail 0013 (FP32) didn't crash but again really low confidences.

I also tried it on a MobileNetSSD_deploy from here: https://github.com/chuanqi305/MobileNet-SSD:

dnn_net = cv2.dnn.readNet("MobileNetSSD_deploy.caffemodel", "MobileNetSSD_deploy.prototxt", "caffe")

# ... same as above

Error:

terminate called after throwing an instance of 'ngraph::NodeValidationFailure'
  what():  Check 'is_valid_permutation(permutation, arg_shape.rank())' failed at /home/pi/openvino/ngraph/src/ngraph/op/transpose.cpp:65:
While validating node 'v1::Transpose Transpose_181(Add_179[0]:f32{1,12,26,26}, Constant_180[0]:i64{4}) -> (dynamic?)':
Permutation AxisVector{0, 3, 3869288809, 1599426646} is not valid for input shape {1,12,26,26}

As for using IECore in Python. Do you have a link to any documentation I could use?

avitial commented 4 years ago

@ark- thanks for the information, you can find the Python API here.

I did a quick test and I see a similar error when using yolov3-tiny (.cfg and .weights) with dnn module:

  what():  Check 'is_valid_permutation(permutation, arg_shape.rank())' failed at /home/pi/openvino/ngraph/src/ngraph/op/transpose.cpp:65:
While validating node 'v1::Transpose Transpose_174(Add_172[0]:f32{1,255,13,13}, Constant_173[0]:i64{4}) -> (dynamic?)':
Permutation AxisVector{0, 3, 56, 0} is not valid for input shape {1,255,13,13}.

Running the very same test on the v2020.4 for Linux* (it has OpenCV==4.4.0), it runs well so I think there could be something missing on our build process for RPI or a possible bug. I'd need to verify this and get back to you.

Also FYI if I run the same model in IR format (converted yolov3-tiny model to IR as described here) model seems to run perfectly fine on RPI+NCS2 from source built environment, thought I'd share it as it could help you in the meantime:

import cv2 as cv

net = cv.dnn.readNet('frozen_darknet_yolov3-tiny_model.xml', 'frozen_darknet_yolov3-tiny_model.bin')

net.setPreferableBackend(cv.dnn.DNN_BACKEND_INFERENCE_ENGINE)  
net.setPreferableTarget(cv.dnn.DNN_TARGET_MYRIAD)

frame = cv.imread('image.jpg')
if frame is None:
   raise Exception('Image not found!')

blob = cv.dnn.blobFromImage(cv.resize(frame, (416, 416)),
            1 / 255, (416, 416), (10, 20, 30), swapRB=True, crop=False)

net.setInput(blob)
out = net.forward()
# not crashing

Regards, Luis

ark- commented 4 years ago

Luis,

Thanks very much for the update and the advice about converting to IR.

I have had a go at freezing my custom model to an IR and I am having errors such as 

terminate called after throwing an instance of 'InferenceEngine::details::InferenceEngineException'
  what():  Check 'PartialShape::merge_into(inputs_shape_scheme, this_input_shape)' failed at /home/pi/openvino/ngraph/src/ngraph/op/concat.cpp:90:
While validating node 'v0::Concat Concat_2423(PRelu_2212[0]:f32{1,256,20,34}, Interpolate_2422[0]:f32{1,256,26,26}) -> (f32{1,512,26,26})':
Argument shapes are inconsistent; they must have the same rank, and must have equal dimension everywhere except on the concatenation axis (axis 1).

I converted to IR the pretrained yolov3 and it worked OK. I've probably missed something in the process so don't worry too much. I'll wait until it is fixed in the RPI build process.

avitial commented 4 years ago

@ark- looks like this issue has been addressed as discussed in https://github.com/opencv/opencv/issues/17954, changes from https://github.com/opencv/opencv/pull/17998 seem to fix this issue.

Also just FYI the pre-built package for Raspbian has been made available in https://download.01.org/opencv/2020/openvinotoolkit/2020.4.

ark- commented 4 years ago

Avitial, thanks for the update. Yeah tested and fixes the issue. Will close 👍