ArmnnDelegate can not assign backends to each layer

[adarsh044]

I am using the armnn delegate by integrating it as AAR to my android project. I declared the armnn class in Java as below :

String[] optionKeys = {"logging-severity","backends", "enable-fast-math","reduce-fp32-to-fp16","number-of-threads"}; String[] optionValues = {"info", "GpuAcc,CpuAcc,CpuRef", "true", "true", "8"}; armnnDelegate = new ArmnnDelegate(optionKeys,optionValues);

Here are some logs :

` TfLiteArmnnDelegate: Created TfLite ArmNN delegate. Initialized TensorFlow Lite runtime. W DEQUANTIZE: not supported by armnn: in validate_arguments src/cpu/kernels/CpuDequantizeKernel.cpp:51: ITensor data type F16 not supported by this kernel W DEQUANTIZE: not supported by armnn: in validate_arguments src/cpu/kernels/CpuDequantizeKernel.cpp:51: ITensor data type F16 not supported by this kernel W DEPTHWISE_CONV2D: not supported by armnn: Backend is not capable of supporting dynamic weights (NonConstWeights) and DepthwiseConvolution2d weights are set as dynamic (non constant). W DEQUANTIZE: not supported by armnn: in validate_arguments src/cpu/kernels/CpuDequantizeKernel.cpp:51: ITensor data type F16 not supported by this kernel W CONV2D: not supported by armnn: Backend is not capable of supporting dynamic weights (NonConstWeights) and Convolution2d weights are set as dynamic (non constant). W DEQUANTIZE: not supported by armnn: in validate_arguments src/cpu/kernels/CpuDequantizeKernel.cpp:51: ITensor data type F16 not supported by this kernel W DEQUANTIZE: not supported by armnn: in validate_arguments src/cpu/kernels/CpuDequantizeKernel.cpp:51: ITensor data type F16 not supported by this kernel W DEQUANTIZE: not supported by armnn: in validate_arguments src/cpu/kernels/CpuDequantizeKernel.cpp:51: ITensor data type F16 not supported by this kernel D Info: ArmnnSubgraph creation D Info: Parse nodes to ArmNN time: 0.01 ms D Info: Optimize ArmnnSubgraph time: 0.25 ms D Info: Load ArmnnSubgraph time: 2.95 ms D Info: Overall ArmnnSubgraph creation time: 3.34 ms D Info: ArmnnSubgraph creation D Info: Parse nodes to ArmNN time: 0.04 ms D Warning: ERROR: output 0 of layer Cast () is of type Quantized 8 bit but its scale parameter has not been set D Warning: WARNING: Layer of type Cast is not supported on requested backend CpuAcc for input data type Float16 and output data type QAsymmU8 (reason: in validate_arguments src/cpu/kernels/CpuCastKernel.cpp:138: Only data_types supported [in] F32 -> [out] QASYMM8, BFLOAT16, F16, S32, U8), falling back to the next backend. D Warning: ERROR: Layer of type Cast is not supported on any preferred backend [CpuAcc ] D Warning: ERROR: output 0 of layer DepthToSpace () is of type Quantized 8 bit but its scale parameter has not been set

E terminating with uncaught exception of type armnn::Exception: TfLiteArmnnDelegate: Exception (Failed to assign a backend to each layer) caught from optimize. `

I get the following error :

terminating with uncaught exception of type armnn::Exception: TfLiteArmnnDelegate: Exception (Failed to assign a backend to each layer) caught from optimize.

Does this issue arise because of wrong declaration? If not what is the issue here?

[matthewsloyanARM]

Hi @adarsh044,

Thank you for getting in touch. Your declaration looks good to me. It looks like there could be a few things with the model that could be causing issues. I see there is dynamic weights for DEPTHWISE_CONV2D which is not supported by Arm NN, however I would expect it to fallback to the TFLite Runtime in this case and all the other cases listed as you are using the Arm NN TFLite Delegate. Would you possibly be able to share the model you are running so I can try and see why it's not falling back for you?

One other thing I noticed is this preferred backend [CpuAcc ], I believe this should list [GpuAcc, CpuAcc, CpuRef ], like you have specified. Can you try it with just one option E.g "CpuRef" in your optionValues to see if it makes any difference? Thanks!

Kind regards,

Matthew

[adarsh044]

Hi @matthewsloyanARM

Thanks for the response to the issue. There are other layers too that give similar not supported warnings like CONV2D. I am attaching a model that I am trying to run. c_dummy_arm_dummy_fp16.zip

I tried changing the backend to just one (both CPU and GPU), issue persists.

Would be helpful if you could check and try running the model , from your side.

Regards,

Adarsh

[TeresaARM]

Tensorflow has not quantized int8 and uint8, that in Arm NN are supported as quantized int8 and uint8 with scale=1 and zero_point=0.

Could you cherry pick the https://review.mlplatform.org/c/ml/armnn/+/9406 and try if that solves your problem? If it does not, please reopen the ticket.

Kindest Regards