Copyright (C) 2023, Axis Communications AB, Lund, Sweden. All Rights Reserved.
Axis network cameras can be used for computer vision applications and can run machine learning models to make inferences. The model to use will depend on your device and your application. This repository contains a collection of different models compatible with Axis cameras and some performance measures (accuracy and speed). Our goal is to keep updating this collection with models for different applications, like object detection or pose estimation. For easy reproduction, we mostly use models that are public and we also share tools to benchmark the models. We have linked the model files and, in case they are public, the ckpt files to continue the training. The values of speed and accuracy are going to be updated with every AXIS OS release.
Note : These are not production-quality models, they are off-the-shelf models used for comparative and demonstration purposes only.
| Platform | Model | TF version | Speed | Accuracy |
|---|---|---|---|---|
| ARTPEC-7 (Q1615 Mk III) | MobilenetV2 (ckpt) | 1 | 4.61 ms | Top 1: 68.9% Top 5: 88.2% |
| ARTPEC-7 (Q1615 Mk III) | MobilenetV2 | 2 | 4.56 ms | Top 1: 69.6% Top 5: 89.1% |
| ARTPEC-7 (Q1615 Mk III) | MobilenetV3 | 2 | 4.70 ms | Top 1: 72.7% Top 5: 91.1% |
| ARTPEC-8 (P1465-LE) | MobilenetV2 (ckpt) | 1 | 9.87 ms | Top 1: 68.8% Top 5: 88.9% |
| ARTPEC-8 (Q1656-LE) | MobilenetV2 (ckpt) | 1 | 5.39 ms | Top 1: 68.8% Top 5: 88.9% |
| CV25 (M3085-V) | MobilenetV2 (ckpt) | 1 | 5.59 ms | Top 1: 66.7% Top 5: 87.1% |
| CV25 (M3085-V) | EfficientNet-Lite0 (ckpt) | 1 | 6.97 ms | Top 1: 71.2% Top 5: 90.3% |
| Platform | Model | Speed | Accuracy |
|---|---|---|---|
| ARTPEC-7 (Q1615 Mk III) | SSD MobileNet v2 | 18.42 ms | mAP: 25.6% |
| ARTPEC-7 (Q1615 Mk III) | SSDLite MobileDet | 31.55 ms | mAP: 32.9% |
| ARTPEC-8 (P1465-LE) | SSD MobilenetV2 | 29.29 ms | mAP: 25.6% |
| ARTPEC-8 (P1465-LE) | SSDLite MobileDet | 40.28 ms | mAP: 32.9% |
| ARTPEC-8 (P1465-LE) | Yolov5n-Artpec8 (ckpt) | 100.04 ms | mAP: 23.5% |
| ARTPEC-8 (Q1656-LE) | SSD MobilenetV2 | 18.89 ms | mAP: 25.6% |
| ARTPEC-8 (Q1656-LE) | SSDLite MobileDet | 30.59 ms | mAP: 32.9% |
| ARTPEC-8 (Q1656-LE) | Yolov5n-Artpec8 (ckpt) | 55.12 ms | mAP: 23.5% |
| ARTPEC-8 (Q1656-LE) | Yolov5s-Artpec8 (ckpt) | 69.53 ms | mAP: 32.3% |
| ARTPEC-8 (Q1656-LE) | Yolov5m-Artpec8 (ckpt) | 94.90 ms | mAP: 37.9% |
Values for AXIS OS 12.1.60.
The auto-test-framework directory holds the code for measuring the speed numbers and automating their update in the repository. For now, the accuracy measures are not included in this pipeline. Apart from that, in the accuracy-test there is the code to measure the accuracy and model_performance_tester.py is a script to measure the speed.
In the case of the automated test framework, the test is done by installing and running an ACAP application on the Axis camera. To know more about how it works, see the larod-test directory.
To get speed measures more easily, you can use the code in model_performance_tester.py. This script connects to the Axis camera via SSH and uses the larod-client to run inferences. It then parses the output, which will be the mean of time the Axis camera spent on the inferences. These inferences are done on randomly generated images. See below how to use the script:
python3 ./model_performance_tester.py \
--model_path <MODEL_PATH> --test_duration <DURATION> \
--chip <CHIP> --device_ip <IP> --device_credentials <USER> <PASS> --camera_port <SSH_PORT>where, <MODEL_PATH> is the path where your .tflite or .bin model is and <DURATION> is the amount of inferences to run. The rest are camera parameters.
This test is done by installing and running an ACAP application on the Axis camera. To know more about how it works, see the accuracy-test directory.