This repository is outdated! Join the open MLPerf workgroup to participate in the development of the next generation of automation workflows for MLPerf benchmarks:
Running MobileNet quantized with TFLite on images preprocessed with OpenCV for MobileNet into rgbf32 fails:
$ ck benchmark program:image-classification-tflite --speed --skip_print_timers \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10 \
--dep_add_tags.images=preprocessed,using-opencv,normalized-for.mobilenet \
--dep_add_tags.weights=mobilenet,quantized
...
Summary:
-------------------------------
Graph loaded in 0.001913s
All images loaded in 0.002327s
All images classified in 0.460941s
Average classification time: 0.043231s
Accuracy top 1: 0.0 (0 of 10)
Accuracy top 5: 0.0 (0 of 10)
--------------------------------
The same works fine with TF-C++:
$ ck benchmark program:image-classification-tf-cpp --speed --skip_print_timers \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10 \
--dep_add_tags.images=preprocessed,using-opencv,normalized-for.mobilenet \
--dep_add_tags.weights=mobilenet,quantized
...
Summary:
-------------------------------
Graph loaded in 0.081466s
All images loaded in 0.002623s
All images classified in 0.528722s
Average classification time: 0.011366s
Accuracy top 1: 0.7 (7 of 10)
Accuracy top 5: 0.9 (9 of 10)
--------------------------------
or with TFLite on images preprocessed using OpenCV into rgb8:
$ ck benchmark program:image-classification-tflite --speed --skip_print_timers \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10 \
--dep_add_tags.images=preprocessed,using-opencv,rgb8 \
--dep_add_tags.weights=mobilenet,quantized
...
Summary:
-------------------------------
Graph loaded in 0.001789s
All images loaded in 0.000663s
All images classified in 0.477427s
Average classification time: 0.043434s
Accuracy top 1: 0.7 (7 of 10)
Accuracy top 5: 0.9 (9 of 10)
--------------------------------
Running MobileNet quantized with TFLite on images preprocessed with OpenCV for MobileNet into rgbf32 fails:
The same works fine with TF-C++:
or with TFLite on images preprocessed using OpenCV into rgb8: