Deploying YOLOv7 / any type of YOLO model (v5,v4,etc)

[philippe-heitzmann]

Support for YOLOv7 / any type of YOLO model (v5,v4,etc)

I was wondering if the ACAP team could please confirm if anyone has managed to deploy any type of YOLO object detector model (v7,v5,v4) to any ARTPEC-8 & aarch64 camera derivates / if this would be at all supported from a hardware perspective for these model types in the case these are quantized to either fp16 / int8?

For reference we are seeing several other issues mentioning quantization to fp16 / int8 would not be supported (#[65], [#62]), and for background our own tests attempting to run fp16 and int8-quantized yolov7 models shows these fail to be correctly loaded to the AXIS runtime via this object-detector-python inference server image per the below logs output:

inference-server_1        | ERROR in Inference: Failed to load model yolov7_fp16_quant.tflite (Could not load model: Asynchronous connection has been closed)

If this would not be possible, could the ACAP team please advise if there would be any other object detection weight files apart from those provided in the object-detector-python MobileNet examples folder that the team would recommend taking a look at? Any pointers on this would be much appreciated if possible, thank you.

[Corallo]

Hello @philippe-heitzmann

The yolo networks usually present several layers that are not well optimized with our devices, and in some cases, not compatible at all. We have seen some people managing successfully to run yolov5, but we have seen also a lot of problems related to them. If latency is not an issue, the easiest seemed to export the model of yolov5 or v8 with their export script to tflite and add the --int8 flag. The issue with that is that you obtain a model quantized per channel, not well optimized for our device, that will run at 1s / inference. We have been working for a bit at creating a way to process the model to obtain a tflite quantized per tensor that reached much higher speed, but we encountered some issue with an accuracy lost, and more investigation was required.

We plan to expand the model-zoo with detection models, but we don't have an ETA yet.

Anyway, we appreciate your feedback, knowing what users are interested in will help us to decide where to focus our attention.

[philippe-heitzmann]

Got it thank you @Corallo!

Understood that certain layers of the YOLO models are not able to be run on ARTPEC-8 chips - would the team be able to please point to which layers these would be if the team would have dug deeper / done research into this? Also for the avoidance of doubt, per your mention that per-channel quantization does not run as efficiently on ARTPEC-8 chips and per tensor runs significantly faster (though at an accuracy tradeoff), would the team therefore recommend exploring models with operations specifically allowing for per-tensor quantization instead of per-channel?

This would be helpful for identifying other models beyond Mobilenet SSD that would be supported on these chips if the team would have any thoughts on this. Thank you for your pointers in this process, super appreciated :)

[Corallo]

I'd recommend this post, it has more information that you might find useful. To answer specifically to your questions:

• The accelerator of A8 speed up sets of Convolution - Batch normalization - ReLU, anything else runs at "cpu speed".
• The most efficient architecture seems to be RegNet, unfortunately there are not many well maintained public implementations.
• The quantization technique is not architecture-dependent, any network in theory can be quantized by tensor or by channels, but rather is framework dependent. The only way to quantize a model by tensor is to use TensorFlow 1, you can find more details on the link that I sent before.

Extra general recommendation: Before digging into finding the best architecture, I'd try locally on your machine with some samples of the frames that you want the model to be able to detect, and check that the architecture actually make a difference for your use case. Sometimes working on the training and the dataset might be more beneficial to a results/effort point of view than chasing the latest architecture.

[Corallo]

Hi again, It is now possible to run Yolov5 on Artpec-8 on devices running Axis OS >= 11.7 See the guide here