Where are benchmarks supporting the claim of being the most performant ML inference engine

[bojeckkk]

As per README:

Arm NN is the most performant machine learning (ML) inference engine for Android and Linux, accelerating ML on Arm Cortex-A CPUs and Arm Mali GPUs.

Are there any benchmarks supporting the statement? From my initial findings it seems to perform much slower (40-50%) than TFLite with XNNPack when it comes to Conv2d nodes (On A52/A53 CPUs). It even seems to be slower than plain TFLite by 10%. I do not have separated test cases for now, but most of the time my model spends on Conv2d.

EDIT: I'm running binaries built for armv7 32-bit

[psyhtest]

Hey @MrSherish, is MLPerf Inference good enough for you? :) We typically find that ArmNN with NEON performs better than TFLite, although not for all models across all platforms.

I will not vouch we use the best build options for TFLite though. At some point (~v2.3?) they introduced support for CMake, but it has been broken since v2.8.

[liamsun2019]

Also curious about the performance on some classical network. As far as I know, some similar frameworks such as TNN, MNN, provide detailed data about performance. I have not yet found such information for ARMNN.

[morgolock]

I'm running binaries built for armv7 32-bit

I'd suggest you move to aarch64 if possible. You will get much better performance if you build for armv8 or newer architectures.

The armv7a code is not as highly optimized as aarch64.

Hope this helps.

[bojeckkk]

Unfortunately I am currently limited to armv7a binaries exclusively.

[bojeckkk]

Thank you @psyhtest , the link you provided should do for now.

[bojeckkk]

The difference is even worse for arm64 using same model - ArmNN performs 9 times slower than XNNPack

[MatthewARM]

Thanks @MrSherish can you attach the model? Something must be broken.