Memory corruption when using OnnxRuntime with OpenVINO on the Intel MyriadX and Raspberry Pi 4B

[maxlytkin]

Hi. I'm creating this ticket on behalf of Stack Overflow user who reported this issue while using OpenVINO toolkit. This component (OpenVINO Execution Provider) is not part of the OpenVINO toolkit. Original thread - https://stackoverflow.com/questions/65253014/memory-corruption-when-using-onnxruntime-with-openvino-on-the-intel-myriadx-and

Describe the bug

I'm trying to run Inference on the Intel Compute Stick 2 (MyriadX chip) connected to a Raspberry Pi 4B using OnnxRuntime and OpenVINO. I have everything set up, the openvino provider gets recognized by onnxruntime and I can see the myriad in the list of available devices.

However, I always get some kind of memory corruption when trying to run inference on the myriad. I'm not sure where this is coming from. If I use the default CPU inference instead of openvino, everythin works fine. Maybe the way I'm creating the Ort::MemoryInfo object is incorrect.

output

Available execution providers:
        CPUExecutionProvider
        OpenVINOExecutionProvider
Available OpenVINO devices:
        MYRIAD
Starting Session
[...]
2020-12-11 13:43:13.962093843 [I:onnxruntime:, openvino_execution_provider.h:124 OpenVINOExecutionProviderInfo] [OpenVINO-EP]Choosing Device: MYRIAD , Precision: FP16
[...]
2020-12-11 13:43:13.972813082 [I:onnxruntime:, capability_2021_1.cc:854 GetCapability_2021_1] [OpenVINO-EP] Model is fully supported by OpenVINO
[...]
Loading data
Running Inference
2020-12-11 13:43:21.838737814 [I:onnxruntime:, sequential_executor.cc:157 Execute] Begin execution
2020-12-11 13:43:21.838892108 [I:onnxruntime:, backend_manager.cc:253 Compute] [OpenVINO-EP] Creating concrete backend for key: MYRIAD|50,28,28,1,|10,|84,10,|84,|120,84,|6,1,5,5,|16,|6,|400,120,|16,6,5,5,|120,|
2020-12-11 13:43:21.838926959 [I:onnxruntime:, backend_manager.cc:255 Compute] [OpenVINO-EP] Backend created for graph OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0
2020-12-11 13:43:21.845913973 [I:onnxruntime:, backend_utils.cc:65 CreateCNNNetwork] ONNX Import Done
malloc(): unsorted double linked list corrupted
Aborted

Here is the code I'm using

#include <iostream>
#include <iomanip>
#include <chrono>
#include <array>
#include <cmath>
#include <MNIST-Loader/MNIST.h>
#include <onnxruntime_cxx_api.h>
#include <core/framework/allocator.h>
#include <ie_core.hpp> //openvino inference_engine

int main()
{
        constexpr const char* modelPath = "/home/pi/data/lenet_mnist.onnx";
        constexpr const char* mnistPath = "/home/pi/data/mnist/";
        constexpr size_t batchSize = 50;

        std::cout << "Available execution providers:\n";
        for(const auto& s : Ort::GetAvailableProviders()) std::cout << '\t' << s << '\n';

        std::cout << "Available OpenVINO devices:\n";
        { //new scope so the core gets destroyed when leaving
                InferenceEngine::Core ieCore;
                for(const auto& d : ieCore.GetAvailableDevices()) std::cout << '\t' << d << '\n';
        }

        // ----------- create session -----------
        std::cout << "Starting Session\n";
        Ort::Env env(ORT_LOGGING_LEVEL_INFO);
        OrtOpenVINOProviderOptions ovOptions;
        ovOptions.device_type = "MYRIAD_FP16";
        Ort::SessionOptions sessionOptions;
        sessionOptions.SetExecutionMode(ORT_SEQUENTIAL);
        sessionOptions.SetGraphOptimizationLevel(ORT_DISABLE_ALL);
        sessionOptions.AppendExecutionProvider_OpenVINO(ovOptions);
        Ort::Session session(env, modelPath, sessionOptions);

        // ----------- load data -----------
        std::cout << "Loading data\n";
        MNIST data(mnistPath);
        const std::array<int64_t, 4> inputShape{batchSize, 28, 28, 1};
        //const auto memoryInfo = Ort::MemoryInfo::CreateCpu(OrtDeviceAllocator, OrtMemTypeCPUInput);
        auto openvinoMemInfo = new OrtMemoryInfo("OpenVINO", OrtDeviceAllocator);
        const Ort::MemoryInfo memoryInfo(openvinoMemInfo);
        std::array<float, batchSize*28*28> batch;
        for(size_t i = 0; i < batchSize; ++i)
        {
                const auto pixels = data.trainingData.at(i).pixelData;
                for(size_t k = 0; k < 28*28; ++k)
                {
                        batch[k + (i*28*28)] = (pixels[k] == 0) ? 0.f : 1.f;
                }
        }
        const Ort::Value inputValues[] = {Ort::Value::CreateTensor<float>(memoryInfo, batch.data(), batch.size(), inputShape.data(), inputShape.size())};

        // ----------- run inference -----------
        std::cout << "Running Inference\n";
        Ort::RunOptions runOptions;
        Ort::AllocatorWithDefaultOptions alloc;
        const char* inputNames [] = { session.GetInputName (0, alloc) };
        const char* outputNames[] = { session.GetOutputName(0, alloc) };
        const auto start = std::chrono::steady_clock::now();
        auto results = session.Run(runOptions, inputNames, inputValues, 1, outputNames, 1);
        const auto end = std::chrono::steady_clock::now();
        std::cout << "\nRuntime: " << std::chrono::duration_cast<std::chrono::milliseconds>(end-start).count() << "ms\n";

        // ----------- print results -----------
        std::cout << "Results:" << std::endl;
        for(Ort::Value& r : results)
        {
                const auto dims = r.GetTensorTypeAndShapeInfo().GetShape();
                for(size_t i = 0; i < dims[0]; ++i)
                {
                        std::cout << "Label: " << data.trainingData.at(i).label << "\tprediction: [ " << std::fixed << std::setprecision(3);
                        for(size_t k = 0; k < dims[1]; ++k) std::cout << r.At<float>({i, k}) << ' ';
                        std::cout << "]\n";
                }
        }
        std::cout.flush();
}

[MaajidKhan]

1. Firstly, I would suggest the user to install the latest OpenVINO version (2021.2) https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html?elq_cid=6757375_ts1611141610167&erpm_id=9826563_ts1611141610167

2. I see that the user is using a MNIST onnx model. The MNIST model from the onnx model zoo is usually a static model with input_shape as 1x1x28x28 and we can't do batching on a static model. I hope the user is using a dynamic onnx model with a input_shape, say Nx1x28x28. Only when using a dynamic model, we can do batch Inferencing. I downloaded the MNIST onnx model from here: https://github.com/onnx/models/tree/master/vision/classification/mnist

3. Say, The user is using a dynamic_model and from the code above, a batch_size=50 is being used which might be a little too much for a MyriadX device(Memory Limitation). I would suggest the user to use a smaller batch_size.

4. currently, OpenVINO-EP will be built as a shared_lib by default. we need to add an additional flag --build_shared_lib while building openvino-ep.

OpenVINO-EP Build Instructions Page: https://github.com/microsoft/onnxruntime/blob/master/BUILD.md#openvino

OpenVINO-EP Docx: https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/OpenVINO-ExecutionProvider.md

so, the build command would look something like this now: command: ./build.sh --config RelWithDebInfo --use_openvino MYRIAD_FP16 --build_shared_lib

After EP is built successfully and we navigate to ../onnxruntime/build/Linux/RelWithDebInfo/ Now, we will have the following shared libraries here. libonnxruntime_providers_openvino.so
libonnxruntime_providers_shared.so
libonnxruntime.so.1.6.0 (This only gets added when you use --build_shared_lib flag while building)

5. There is no need to create a seperate Ort::MemoryInfo object specific for openvino. As you are already using the OpenVINO c++ API ( sessionOptions.AppendExecutionProvider_OpenVINO(ovOptions); ), it would be taken care of.

6. For your reference, Attaching a working cpp sample code for mnist.onnx model (static model) which I got from the model zoo. mnist_openvino_ep_sample.txt

Adding steps I used to compile on Linux (ubuntu os):

step 1: command to compile (compiling from this path: ./onnxruntime/build/Linux/RelWithDebInfo/) g++ -o run_mnist mnist_openvino_sample.cpp -I ../../../include/onnxruntime/core/session/ -L ./ -lonnxruntime_providers_openvino -lonnxruntime_providers_shared -lonnxruntime

step 2: ./run_mnist

Prior to compiling, make sure all the three shared libraries mentioned at the top are copied to this location. /usr/lib/x86_64-linux-gnu/

Here's the output from MNIST model using MYRIADX device with OpenVINO-EP:

I have extra prints in the picture, because I was running in debug mode.

Let me know, if you need any more information or if you run into issue again.