Unable to optimize a network with PyArmNN

[fr4wanglei]

Hi, I am trying to optimize a network on an aarch64 machine and got the this error: RuntimeError: Dimension index: 0 not specified. Tensor shape not inferred yet. at function CheckDimensionSpecified

ArmNN version: 22.08 TF version: 2.5.1 python3.8 ubuntu1804

I trained the model with yolov5 and export to fp16.tflite

Any suggestion?

Thank you

[morgolock]

Hi @fr4wanglei

Could you please share with us the command you used to run the model?

[fr4wanglei]

Code:

import onnxruntime as ort
import pyarmnn as ann

from utils import *

class OnnxDetect:
    def __init__(self, args):
        self.opt = args

        parser = ann.ITfLiteParser()
        network = parser.CreateNetworkFromBinaryFile(self.opt.weights)
        options = ann.CreationOptions()
        runtime = ann.IRuntime(options)
        preferredBackends = [ann.BackendId('CpuAcc'), ann.BackendId('CpuRef')]
        opt_network, messages = ann.Optimize(network, preferredBackends, runtime.GetDeviceSpec(), ann.OptimizerOptions())

        self.session = ort.InferenceSession(self.opt.weights, providers=['CPUExecutionProvider'])

I developed real-time detection of rtsp video streams on the arm development board, and started using onnxruntime, but its inference speed is very slow in cpu mode, and each frame takes about 1.5s, which is completely unusable (only required on x86_64 servers 150ms).

Development Board Hardware Specifications： CPU： RK3399Pro，Dual-core Cortex-A72 + Quad-core Cortex-A53 GPU： ARMMali-T860MP4 NPU： 1920 lnt8 MAC operations per cycle；64 FP16 MAC operations per cycle;192 Int16 MAC operations per cycle

Unable to use onnxruntime gpu, so I'm going to try armnn, when I try to create armnn runtime, network = parser.CreateNetworkFromBinaryFile(self.args.weights) I get an error

sorry, my english is not good, i use google translate to communicate with you.

[FrancisMurtagh-arm]

Hi @fr4wanglei,

Can you confirm if you are using the OnnxParser or the TfLiteParser?

If it is TfLiteParser; we recently merged a patch related to yoloV5 which may fix your issue: https://review.mlplatform.org/c/ml/armnn/+/8824

Also is this the version of yolov5 you are using? https://tfhub.dev/neso613/lite-model/yolo-v5-tflite/tflite_model/1

Regards, Francis.

[FrancisMurtagh-arm]

Closing due to lack of activity, please reopen if the issue persists.

Regards, Francis.