search

openvinotoolkit / openvino

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference
https://docs.openvino.ai
Apache License 2.0
7.34k stars 2.29k forks source link

Crash on Raspberry Pi 4 (32 bit) with NCS2 #1503

Closed ericvanbuhler closed 4 years ago

ericvanbuhler commented 4 years ago

Versions

Background

I'm having issues performing inference using OpenVINO with OpenCV's DNN backend. The models I've tried are two mobilenet SSD models:

  1. https://github.com/chuanqi305/MobileNet-SSD
  2. https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md

I saw this error on OpenVINO 2020.3.0 with OpenCV 4.3.0-openvino, but found issue https://github.com/NervanaSystems/ngraph/issues/4794 which looked like it might be related, and saw that the fix (https://github.com/NervanaSystems/ngraph/pull/4745) was going into OpenVINO 2020.4 so I decided to wait and give it a try. Unfortunately the issue persists in 2020.4. I see a similar issue posted recently (https://github.com/openvinotoolkit/openvino/issues/1477) but the crash signature seemed different enough that I decided to add mine as well.

I compiled OpenVINO and OpenCV using nearly the same build flags (changing only architecture specific ones) for x86_64 and aarch64 and both of those work correctly. Also, I was using a similar build process with OpenVINO 2019_R3.1 and OpenCV 4.1.2 and those models were working on the NCS2.

Issue

I see the following crash when I perform an inference with the first model above:

terminate called after throwing an instance of 'ngraph::NodeValidationFailure'
  what():  Check 'is_valid_permutation(permutation, arg_shape.rank())' failed at /opencv-build/build/openvino/ngraph/src/ngraph/op/transpose.cpp:65:
While validating node 'v1::Transpose Transpose_421(Add_408[0]:f32{1,24,1,1}, Constant_420[0]:i64{4}) -> (dynamic?)':
Permutation AxisVector{0, 3, 1400398157, 1350070082} is not valid for input shape {1,24,1,1}

I see a similar error with the second model as well.

Reproduce

I built OpenVINO with the following CMake flags:

-DCMAKE_BUILD_TYPE=Release
-DENABLE_SSE42=OFF
-DTHREADING=SEQ
-DENABLE_GNA=OFF
-DENABLE_MKL_DNN=OFF
-DENABLE_CLDNN=OFF
-DENABLE_SAMPLES=OFF
-DENABLE_OPENCV=OFF
-DENABLE_TESTS=OFF
-DNGRAPH_UNIT_TEST_OPENVINO_ENABLE=OFF
-DNGRAPH_ONNX_IMPORT_ENABLE=OFF

and OpenCV with the following CMake flags:

-DCMAKE_INSTALL_PREFIX=/usr
-DOPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-4.4.0/modules
-DCPACK_BINARY_DEB:BOOL=ON
-DOPENCV_VCSVERSION=4.4.0-openvino
-DCMAKE_BUILD_TYPE=RELEASE
-DBUILD_TESTS=OFF
-DOPENCV_ENABLE_NONFREE=OFF
-DINSTALL_PYTHON_EXAMPLES=OFF
-DBUILD_EXAMPLES=OFF
-DOPENCV_PYTHON_INSTALL_PATH=/usr/local/lib/python3.6/dist-packages
-DENABLE_NEON=ON
-DCPU_BASELINE="NEON"
-DWITH_INF_ENGINE=ON
-DINF_ENGINE_LIB_DIRS="/usr/local/deployment_tools/inference_engine/lib/armv7l/"
-DINF_ENGINE_INCLUDE_DIRS="/usr/local/deployment_tools/inference_engine/include"
-DINF_ENGINE_RELEASE=2020040000
-DOPENCV_GENERATE_PKGCONFIG=ON
-DENABLE_CXX11=ON

Then ran net.forward() using the models listed above.

Regards, Eric

ilyachur commented 4 years ago

Hi

You got the error message "Permutation AxisVector{0, 3, 1400398157, 1350070082} is not valid for input shape {1,24,1,1}" because we cannot validate graph. How did you get IR files? Do you use OpenCV to execute?

ericvanbuhler commented 4 years ago

@ilyachur Thanks for taking a look! I used OpenCV with OpenVINO as the inference engine backend, and used the files provided for each model (caffe for 1, tensorflow for 2) as input to cv2.dnn.readNet(). I believe the conversion to IR happens at runtime, but I'm not sure where that happens.

ilyachur commented 4 years ago

@dkurt could you comment, how OpenCV generates ngraph function?

dkurt commented 4 years ago

I'd like to recommend to wait for publication of 2020.4 archive of OpenVINO for Raspberry Pi.

ericvanbuhler commented 4 years ago

@dkurt What's the difference between what I've built and the archive for Raspberry Pi? I'm happy to wait, but I just want to make sure that I can still use it in the same way I am now. Most importantly, that I can still use it as the inference engine backend of OpenCV.

dkurt commented 4 years ago

OpenVINO: 2020.4 OpenCV: 4.4.0-openvino

@ericvanbuhler, have you used exactly commits under these tags?

https://github.com/openvinotoolkit/openvino/tree/2020.4
https://github.com/opencv/opencv/tree/4.4.0-openvino

ericvanbuhler commented 4 years ago

@dkurt Yes, I believe so. Here are the steps I did to check out OpenVINO and OpenCV:

git clone https://github.com/openvinotoolkit/openvino.git
cd openvino
git checkout 2020.4
git submodule update --init --recursive
<build OpenVINO>
...

wget -O opencv.zip https://github.com/opencv/opencv/archive/4.4.0-openvino.zip
unzip opencv.zip
wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.4.0.zip
unzip opencv_contrib.zip
<build OpenCV>
dkurt commented 4 years ago

@ericvanbuhler , sorry for late response. I were able to reproduce a problem:

terminate called after throwing an instance of 'ngraph::NodeValidationFailure'
  what():  Check 'is_valid_permutation(permutation, arg_shape.rank())' failed at /openvino/ngraph/src/ngraph/op/transpose.cpp:65:
While validating node 'v1::Transpose Transpose_181(Add_179[0]:f32{1,12,19,19}, Constant_180[0]:i64{4}) -> (dynamic?)':
Permutation AxisVector{0, 3, 1952804193, 32} is not valid for input shape {1,12,19,19}

Aborted

this is my Dockerfile: https://gist.github.com/dkurt/585f886a000d3afade62f66c25640148

OpenCV: https://github.com/opencv/opencv/commit/a2f7ef9d210e5be5741c6e3109c233672534a1ed OpenVINO: https://github.com/openvinotoolkit/openvino/commit/6c3b7ee8cab5b6d7cd3c82f73cae32163a8c9472

dkurt commented 4 years ago

I've decided to setup my own build and test nightly infrastructure for OpenVINO on Raspberry Pi with NCS2: https://github.com/dkurt/vineyard. For now there are failed tests including mentioned error:

[ RUN      ] Test_ONNX_layers.ReduceL2/0, where GetParam() = NGRAPH/MYRIAD
unknown file: Failure
C++ exception with description "Check 'is_valid_permutation(permutation, arg_shape.rank())' failed at /openvino/ngraph/src/ngraph/op/transpose.cpp:65:
While validating node 'v1::Transpose Transpose_482932(Multiply_482930[0]:f32{1,3,2,4}, Constant_482931[0]:i64{4}) -> (dynamic?)':
Permutation AxisVector{0, 3, 24, 30177192} is not valid for input shape {1,3,2,4}
" thrown in the test body.
ericvanbuhler commented 4 years ago

@dkurt Thanks for your work on this issue! Your build and test infrastructure will help resolve this and hopefully anything else that comes up on RPi + OpenVINO + NCS2. It's a really important use case for me!

dkurt commented 4 years ago

@ericvanbuhler, https://github.com/opencv/opencv/pull/17998 going to fix it (12 failed tests against 31 - the rest are minor ones related to deprecated IR version).

ark- commented 4 years ago

I'll give it a shot in the morning, thanks for sharing the PR.

ark- commented 4 years ago

@ericvanbuhler, opencv/opencv#17998 going to fix it (12 failed tests against 31 - the rest are minor ones related to deprecated IR version).

Thanks I have tried this now. I get a cmake error if I try to compile with OpenCV Contrib but thats to do with compiling unmatched branches, I'll just wait until the fix has made its way to master before I use contrib again.

Having said that, I can now successfully forward darknet models 👍 Thanks!

Edit: Removed the issue I had, I figured out the problem was I wasn't passing the output layers to the dnn.forward()

ericvanbuhler commented 4 years ago

@dkurt Awesome! I'll try it out as well this week.