This repository is outdated! Join the open MLPerf workgroup to participate in the development of the next generation of automation workflows for MLPerf benchmarks:
I've created a copy of the script we used for benchmarking Image Classification models for the Closed division of MLPerf Inference v0.5. Below are some suggestions for updating this script for MLPerf Inference v0.7.
[x] On a platform like Firefly-RK3399, we support 3 workflow variants (implementations/backends): TFLite, ArmNN with Neon support, ArmNN with OpenCL support. For v0.5, we modified the script to specify which variant is "active" and relaunched. Near the submission deadline, we introduced iteration over implementations and backends into the audit.sh script. Now we should do similarly for the run.sh script. Bonus points for spotting other improvements based on the audit.sh script or the run.sh script for the Open division!
[x] In addition to rpi4, add other platforms that do not support ArmNN with OpenCL such as Jetson TX1 (tx1). In fact, perhaps write a function that checks whether the given platform is in an OpenCL "blacklist"?
[x] MobileNet has been dropped, so this leaves ResNet only. For simplicity, we will only support one preprocessing method (using OpenCV). Therefore, we should change this:
[x] We should update the library versions: currently, ArmNN v20.05 and TFLite v1.15.3 are the latest supported (although we may be able to support TFLite 2.x?)
[x] In principle, the difference between Closed and Open division submissions will be primarily in the list of models to use: ResNet only for the Closed; all MobileNet variants for the Open. Having a unified script would be great!
I've created a copy of the script we used for benchmarking Image Classification models for the Closed division of MLPerf Inference v0.5. Below are some suggestions for updating this script for MLPerf Inference v0.7.
[x] On a platform like Firefly-RK3399, we support 3 workflow variants (implementations/backends): TFLite, ArmNN with Neon support, ArmNN with OpenCL support. For v0.5, we modified the script to specify which variant is "active" and relaunched. Near the submission deadline, we introduced iteration over implementations and backends into the
audit.shscript. Now we should do similarly for therun.shscript. Bonus points for spotting other improvements based on theaudit.shscript or therun.shscript for the Open division![x] In addition to
rpi4, add other platforms that do not support ArmNN with OpenCL such as Jetson TX1 (tx1). In fact, perhaps write a function that checks whether the given platform is in an OpenCL "blacklist"?[x] MobileNet has been dropped, so this leaves ResNet only. For simplicity, we will only support one preprocessing method (using OpenCV). Therefore, we should change this:
to
[x] We should update the library versions: currently, ArmNN v20.05 and TFLite v1.15.3 are the latest supported (although we may be able to support TFLite 2.x?)
[x] In principle, the difference between Closed and Open division submissions will be primarily in the list of models to use: ResNet only for the Closed; all MobileNet variants for the Open. Having a unified script would be great!
[x] Another idea for unification. We typically have two variants of a workflow: with LoadGen and without LoadGen support e.g.
program:image-classification-tflite-loadgenandprogram:image-classification-tflite. For testing purposes, we should be able to specify which program variants to use as e.g. elsewhere. This choice can manifest itself e.g. here.