search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.13k stars 2.85k forks source link

Raspberry Pi 4 Myriad VPU support #8285

Open rhysdg opened 3 years ago

rhysdg commented 3 years ago

Hi there! First of all hello! I've been using onnx for a little while now and I'm absolutely loving being able to fling about models across a variety of platforms, especially loving being able to cram a chunky model into my AGX Xavier with onnx - TensorRT, so thank you for all the hard work!

Issue :

I was hoping someone could clarify the status of Myriad VPU (Neural Compute Stick 2) support with the raspberry Pi platform? I see that there's support for building from source, and docker in your documentation here and I've also seen mention of cross-compiling. I don't seem to see any clear examples of bulding with the ability to employ Myriad as a device within a RPI runtime however. It'd be much appreciated if you could point me in the right direction as to a way to go about this, or as to the status of any potential incoming updates.

Current runtime/device :

Thanks for your time and hope you're all well!

jywu-msft commented 3 years ago

This has been tested in the distant past and it was working back then. Unfortunately, it's not currently documented and hasn't been validated on current releases. As I recall, the gist of it is you'll need to download the openvino toolkit for raspberry pi. https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_raspbian.html and then you'll need to cross-compile OnnxRuntime for ARM32 and enable OpenVINO Execution Provider. Unfortunately there isn't currently a dockerfile that has this enabled, but you might be able to start from https://github.com/microsoft/onnxruntime/blob/master/dockerfiles/README.md#arm-32v7 and modify it to add openvino build flags. (--use_openvino MYRIAD_FP16) I will connect with the folks from Intel next week to see if they are willing to test/validate and publish a tutorial for this.

znsoftm commented 3 years ago

There's a way to compile openvino aarch64 on Raspberry pi 4, so is it possible onnxruntime works with openvino on aarch64?

rhysdg commented 3 years ago

Awesome! Cheers guys. It's good to know at least that there has been success in the past with this. I'll head down the cross-compilation rabbit hole and report back if I come up with something that can be reproduced!.

rhysdg commented 3 years ago

Alright we're I'm at right now is the following:

jywu-msft commented 3 years ago

adding some folks from intel for their input. +@smkarlap , @suryasidd, @maajidkhan

MaajidKhan commented 3 years ago

@rhysdg --use_openvino MYRIAD_FP16 returns a non-zero code: 1.

I think onnxruntime throws this error as it is not able to find openvino package which has the openvino libs.

Did you follow the steps mentioned here to install openvino for Raspbian os. https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_raspbian.html#install-package

Raspbian OS OpenVINO package Tar file can be downloaded from: https://storage.openvinotoolkit.org/repositories/openvino/packages/2021.4/

-> Install the OpenVINO Toolkit components -> Install the External software dependencies -> Set the Environment Variables -> You need to configure your NCS2 stick by adding USB rules as well

This completes the openvino installation steps for Raspbian OS environment.

But all of these steps are for local build. In your case, you need to replicate the above steps by configuring them at the top of this dockerfile (https://github.com/microsoft/onnxruntime/blob/master/dockerfiles/README.md#arm-32v7)

You can use this dockerfile as reference of how to set the environment for openvino libs https://github.com/intel/onnxruntime/blob/master/dockerfiles/Dockerfile.openvino

only these ENV settings would suffice: (Inference Engine libs + ngraph libs)

ENV INTEL_OPENVINO_DIR=/opt/intel/openvino_2021 ENV InferenceEngine_DIR=${INTEL_OPENVINO_DIR}/deployment_tools/inference_engine/share ENV IE_PLUGINS_PATH=${INTEL_OPENVINO_DIR}/deployment_tools/inference_engine/lib/armv7l ENV LD_LIBRARY_PATH=$INTEL_OPENVINO_DIR/deployment_tools/ngraph/lib:${IE_PLUGINS_PATH}:${LD_LIBRARY_PATH} ENV ngraph_DIR=${INTEL_OPENVINO_DIR}/deployment_tools/ngraph/cmake

Then as final step, you can add openvino build flag --use_openvino MYRIAD_FP16 along with onnxruntime build flags in the same dockerfile.

Note: I don't have a Raspberry pi 4 device to try this out myself and I haven't tried this out myself in the past. Please let us know if the above steps could help you to generate the wheel for onnxruntime-openvino.

rhysdg commented 3 years ago

Aha Wonderful! I'll give it a shot at the start of the week and report back. Thanks for the detailed response @MaajidKhan!

MaajidKhan commented 3 years ago

@rhysdg Hi, I got a Raspberry pi 4 Model B recently and I was able to reproduce the issue from my end. I was trying out many work arounds to get openvino Execution provider working with NCS2 stick on Raspberry pi with Raspbian OS. I was able to find some success with one solution.

Raspberry pi details: I have the latest Raspberry pi 4 Model B with 8GB of ram and 4 cores and it’s Arm32. I have SD card of 128 GB and the NCS2 stick(512 MB one) plugged in.

Steps: 1.Installed Rasbian Buster 32 bit OS on the Raspberry pi 4.

  1. Installed openvino rasbian package and other NCS2 drivers. https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_raspbian.html

OpenVINO Raspbian package is available at: https://storage.openvinotoolkit.org/repositories/openvino/packages/

Follow the steps from the above link. I was able to successfully infer/run the model using Native openvino on NCS2 stick.

3. Now building OpenVINO Execution Provider for ONNXRuntime for ARM with Rasbian OS. I tried many workarounds, but this is the one where I found some success.

Native compilation on ARM. steps: 1. Open a New Terminal on Raspberrypi and source openvino source /opt/intel/openvino_2021/bin/setupvars.sh

2.

git clone https://github.com/microsoft/onnxruntime.git

cd onnxruntime

./build.sh --build --update --config Release --use_openvino MYRIAD_FP16 --build_shared_lib --build_wheel --parallel --skip_tests --arm

3. copy the whl from the /dist folder.

cd build/Linux/Release/dist

4. pip install the wheel. pip3 install onnxruntime_openvino-1.8.2-cp37-cp37m-linux_armv7l

Try running some python sample with openvino Execution Provider with MYRIAD_FP16 setting (NCS2, VPU)

Note: I was able to run both python sample and cpp samples on NCS2. But just before the application exits, it throws an error and a segfault at times. we will investigate this more further.

Error: python_app_error onnxruntime_perf_test_error

But I would like you to try this and see if you can get till here.

Also attaching the pip wheel which I was able to generate. You can try installing it directly on the Raspberry pi and give it a try. [Uploading onnxruntime_openvino-1.8.2-cp37-cp37m-linux_armv7l.whl…]()

MaajidKhan commented 2 years ago

@rhysdg Please close this ticket if you were able to resolve this issue by trying the steps in the comment above. Thanks.

rhysdg commented 2 years ago

Hi there @MaajidKhan my apologies for the late reply. I'm afraid I've been unable to confirm that I can reproduce those steps as my workload has seen a bit of an increase. I'll schedule in some time to run through it next week however! Great to see there's a partial resolution to the problem and I appreciate your efforts!

drux007 commented 2 years ago

@rhysdg Hi, I got a Raspberry pi 4 Model B recently and I was able to reproduce the issue from my end. I was trying out many work arounds to get openvino Execution provider working with NCS2 stick on Raspberry pi with Raspbian OS. I was able to find some success with one solution.

Raspberry pi details: I have the latest Raspberry pi 4 Model B with 8GB of ram and 4 cores and it’s Arm32. I have SD card of 128 GB and the NCS2 stick(512 MB one) plugged in.

Steps: 1.Installed Rasbian Buster 32 bit OS on the Raspberry pi 4. 2. Installed openvino rasbian package and other NCS2 drivers. https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_raspbian.html

OpenVINO Raspbian package is available at: https://storage.openvinotoolkit.org/repositories/openvino/packages/

Follow the steps from the above link. I was able to successfully infer/run the model using Native openvino on NCS2 stick.

3. Now building OpenVINO Execution Provider for ONNXRuntime for ARM with Rasbian OS. I tried many workarounds, but this is the one where I found some success.

Native compilation on ARM. steps: 1. Open a New Terminal on Raspberrypi and source openvino source /opt/intel/openvino_2021/bin/setupvars.sh

2.

git clone https://github.com/microsoft/onnxruntime.git

cd onnxruntime

./build.sh --build --update --config Release --use_openvino MYRIAD_FP16 --build_shared_lib --build_wheel --parallel --skip_tests --arm

3. copy the whl from the /dist folder.

cd build/Linux/Release/dist

4. pip install the wheel. pip3 install onnxruntime_openvino-1.8.2-cp37-cp37m-linux_armv7l

Try running some python sample with openvino Execution Provider with MYRIAD_FP16 setting (NCS2, VPU)

Note: I was able to run both python sample and cpp samples on NCS2. But just before the application exits, it throws an error and a segfault at times. we will investigate this more further.

Error: python_app_error onnxruntime_perf_test_error

But I would like you to try this and see if you can get till here.

Also attaching the pip wheel which I was able to generate. You can try installing it directly on the Raspberry pi and give it a try. Uploading onnxruntime_openvino-1.8.2-cp37-cp37m-linux_armv7l.whl…

Hi @MaajidKhan, I was following your instructions, but I cannot build it successfully with the latest release.

It seems like an issue is introduced in v1.9.0.

I created a separate issue for it. Please see #9659.

drux007 commented 2 years ago

Hi guys,

I managed to successfully build onnxruntime (v1.8.2) with OpenVINO support for a Raspberry Pi 4.

When I try to run tinyYOLOv2 on RPi4 with Intel NCS2 (MYRIAD_FP16) I get corrupted double-linked list error.

Full log:

python3 tiny_yolov2_obj_detection_sample.py 
2021-11-10 14:25:54.383865907 [I:onnxruntime:, inference_session.cc:230 operator()] Flush-to-zero and denormal-as-zero are off
2021-11-10 14:25:54.383982388 [I:onnxruntime:, inference_session.cc:237 ConstructorCommon] Creating and using per session threadpools since use_per_session_threads_ is true
2021-11-10 14:25:54.935694136 [W:onnxruntime:, graph.cc:1074 Graph] Initializer scalerPreprocessor_scale appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.935831116 [W:onnxruntime:, graph.cc:1074 Graph] Initializer scalerPreprocessor_bias appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.935900449 [W:onnxruntime:, graph.cc:1074 Graph] Initializer convolution_W appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.935969968 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_scale appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936033856 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_B appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936099930 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_mean appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936165189 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_variance appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936229484 [W:onnxruntime:, graph.cc:1074 Graph] Initializer convolution1_W appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936292854 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_scale1 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936356021 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_B1 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936418280 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_mean1 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936481427 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_variance1 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936543353 [W:onnxruntime:, graph.cc:1074 Graph] Initializer convolution2_W appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936606834 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_scale2 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936669000 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_B2 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936731833 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_mean2 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936796592 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_variance2 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936859814 [W:onnxruntime:, graph.cc:1074 Graph] Initializer convolution3_W appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936921999 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_scale3 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.936983924 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_B3 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937045405 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_mean3 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937107924 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_variance3 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937170645 [W:onnxruntime:, graph.cc:1074 Graph] Initializer convolution4_W appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937234293 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_scale4 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937296089 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_B4 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937358274 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_mean4 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937419625 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_variance4 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937482606 [W:onnxruntime:, graph.cc:1074 Graph] Initializer convolution5_W appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937544810 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_scale5 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937607087 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_B5 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937668883 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_mean5 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937731235 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_variance5 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937792938 [W:onnxruntime:, graph.cc:1074 Graph] Initializer convolution6_W appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937856086 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_scale6 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937917437 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_B6 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.937979807 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_mean6 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.938045788 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_variance6 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.938108547 [W:onnxruntime:, graph.cc:1074 Graph] Initializer convolution7_W appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.938170528 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_scale7 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.938234009 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_B7 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.938295990 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_mean7 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.938364249 [W:onnxruntime:, graph.cc:1074 Graph] Initializer BatchNormalization_variance7 appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.938426064 [W:onnxruntime:, graph.cc:1074 Graph] Initializer convolution8_W appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.938487878 [W:onnxruntime:, graph.cc:1074 Graph] Initializer convolution8_B appears in graph inputs and will not be treated as constant value/weight. This may prevent some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2021-11-10 14:25:54.960560559 [I:onnxruntime:Default, openvino_execution_provider.h:127 OpenVINOExecutionProviderInfo] [OpenVINO-EP]Choosing Device: MYRIAD , Precision: FP16
2021-11-10 14:25:54.961264777 [I:onnxruntime:Default, bfc_arena.cc:25 BFCArena] Creating BFCArena for OpenVINO with following configs: initial_chunk_size_bytes: 1048576 max_dead_bytes_per_chunk: 134217728 initial_growth_chunk_size_bytes: 2097152 memory limit: 4294967295 arena_extend_strategy: 0
2021-11-10 14:25:54.961350740 [V:onnxruntime:Default, bfc_arena.cc:61 BFCArena] Creating 21 bins of max chunk size 256 to 268435456
2021-11-10 14:25:54.961456443 [I:onnxruntime:, inference_session.cc:1141 Initialize] Initializing session.
2021-11-10 14:25:54.961526035 [I:onnxruntime:, inference_session.cc:1178 Initialize] Adding default CPU execution provider.
2021-11-10 14:25:54.971539270 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 0
2021-11-10 14:25:54.972725282 [I:onnxruntime:Default, capability.cc:110 Execute] [OpenVINO-EP] Model is fully supported by OpenVINO
2021-11-10 14:25:55.490906610 [I:onnxruntime:Default, backend_manager.cc:85 BackendManager] [OpenVINO-EP] Model has symbolic input dims. Defering backend initialization
2021-11-10 14:25:55.491716902 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 0
2021-11-10 14:25:55.492383101 [V:onnxruntime:, inference_session.cc:909 TransformGraph] Node placements
2021-11-10 14:25:55.492422212 [V:onnxruntime:, inference_session.cc:911 TransformGraph] All nodes have been placed on [OpenVINOExecutionProvider].
2021-11-10 14:25:55.492607544 [V:onnxruntime:, session_state.cc:77 CreateGraphInfo] SaveMLValueNameIndexMapping
2021-11-10 14:25:55.492736136 [V:onnxruntime:, session_state.cc:123 CreateGraphInfo] Done saving OrtValue mappings.
2021-11-10 14:25:55.493004042 [I:onnxruntime:, session_state_utils.cc:142 SaveInitializedTensors] Saving initialized tensors.
2021-11-10 14:25:55.493097004 [I:onnxruntime:Default, bfc_arena.cc:305 AllocateRawInternal] Extending BFCArena for OpenVINO. bin_num:16 (requested) num_bytes: 18874368 (actual) rounded_bytes:18874368
2021-11-10 14:25:55.493178300 [I:onnxruntime:Default, bfc_arena.cc:185 Extend] Extended allocation by 33554432 bytes.
2021-11-10 14:25:55.493214374 [I:onnxruntime:Default, bfc_arena.cc:188 Extend] Total allocated bytes: 33554432
2021-11-10 14:25:55.493247226 [I:onnxruntime:Default, bfc_arena.cc:191 Extend] Allocated memory at 0xa3ed7020 to 0xa5ed7020
2021-11-10 14:25:55.597519002 [I:onnxruntime:Default, bfc_arena.cc:305 AllocateRawInternal] Extending BFCArena for OpenVINO. bin_num:0 (requested) num_bytes: 256 (actual) rounded_bytes:256
2021-11-10 14:25:55.597629483 [I:onnxruntime:Default, bfc_arena.cc:185 Extend] Extended allocation by 33554432 bytes.
2021-11-10 14:25:55.597666520 [I:onnxruntime:Default, bfc_arena.cc:188 Extend] Total allocated bytes: 67108864
2021-11-10 14:25:55.597699168 [I:onnxruntime:Default, bfc_arena.cc:191 Extend] Allocated memory at 0x9d2d3020 to 0x9f2d3020
2021-11-10 14:25:55.609115877 [I:onnxruntime:Default, bfc_arena.cc:305 AllocateRawInternal] Extending BFCArena for OpenVINO. bin_num:17 (requested) num_bytes: 37748736 (actual) rounded_bytes:37748736
2021-11-10 14:25:55.609222672 [I:onnxruntime:Default, bfc_arena.cc:185 Extend] Extended allocation by 67108864 bytes.
2021-11-10 14:25:55.609259080 [I:onnxruntime:Default, bfc_arena.cc:188 Extend] Total allocated bytes: 134217728
2021-11-10 14:25:55.609291376 [I:onnxruntime:Default, bfc_arena.cc:191 Extend] Allocated memory at 0x992d2020 to 0x9d2d2020
2021-11-10 14:25:55.848977000 [I:onnxruntime:, session_state_utils.cc:263 SaveInitializedTensors] Done saving initialized tensors
2021-11-10 14:25:55.856980452 [I:onnxruntime:, inference_session.cc:1329 Initialize] Session successfully initialized.
[ WARN:0] global ../opencv/modules/videoio/src/cap_gstreamer.cpp (1044) open OpenCV | GStreamer warning: unable to query duration of stream
[ WARN:0] global ../opencv/modules/videoio/src/cap_gstreamer.cpp (1081) open OpenCV | GStreamer warning: Cannot query video position: status=1, value=0, duration=-1
2021-11-10 14:25:56.292230657 [I:onnxruntime:, sequential_executor.cc:157 Execute] Begin execution
2021-11-10 14:25:56.292558395 [I:onnxruntime:Default, backend_manager.cc:253 Compute] [OpenVINO-EP] Creating concrete backend for key: MYRIAD|1,3,416,416,|125,|125,1024,1,1,|1024,|1024,|64,32,3,3,|1024,1024,3,3,|32,|256,|16,|64,|1024,|256,|3,1,1,|256,128,3,3,|16,3,3,3,|64,|1024,|16,|16,|1024,|64,|32,|32,|16,|1,|512,|32,|128,64,3,3,|128,|128,|128,|128,|32,16,3,3,|256,|256,|512,256,3,3,|512,|1024,|512,|512,|1024,|1024,|64,|1024,512,3,3,|
2021-11-10 14:25:56.292609414 [I:onnxruntime:Default, backend_manager.cc:255 Compute] [OpenVINO-EP] Backend created for graph OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0
2021-11-10 14:25:57.810726080 [I:onnxruntime:Default, backend_utils.cc:117 CreateCNNNetwork] ONNX Import Done
corrupted double-linked list
Aborted

tinyYOLOv2 example works on my laptop (Ubuntu 20.04 x86_64), but it does not work on a RPi.

The same goes for my custom model: it works on my laptop, but it doesn't on a RPi. In this case, I get corrupted double-linked list or Segmentation fault error.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.