ctuning / ck-mlperf

This repository is outdated! Join the open MLPerf workgroup to participate in the development of the next generation of automation workflows for MLPerf benchmarks:
https://bit.ly/mlperf-edu-wg
BSD 3-Clause "New" or "Revised" License
32 stars 23 forks source link

TFLite MobileNet quantized (int8) fails with OpenCV preprocessing for MobileNet (rgbf32) #40

Open psyhtest opened 5 years ago

psyhtest commented 5 years ago

Running MobileNet quantized with TFLite on images preprocessed with OpenCV for MobileNet into rgbf32 fails:

$ ck benchmark program:image-classification-tflite --speed --skip_print_timers \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10 \
--dep_add_tags.images=preprocessed,using-opencv,normalized-for.mobilenet \
--dep_add_tags.weights=mobilenet,quantized
...
Summary:
-------------------------------
Graph loaded in 0.001913s
All images loaded in 0.002327s
All images classified in 0.460941s
Average classification time: 0.043231s
Accuracy top 1: 0.0 (0 of 10)
Accuracy top 5: 0.0 (0 of 10)
--------------------------------

The same works fine with TF-C++:

$ ck benchmark program:image-classification-tf-cpp --speed --skip_print_timers \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10 \
--dep_add_tags.images=preprocessed,using-opencv,normalized-for.mobilenet \
--dep_add_tags.weights=mobilenet,quantized
...
Summary:
-------------------------------
Graph loaded in 0.081466s
All images loaded in 0.002623s
All images classified in 0.528722s
Average classification time: 0.011366s
Accuracy top 1: 0.7 (7 of 10)
Accuracy top 5: 0.9 (9 of 10)
--------------------------------

or with TFLite on images preprocessed using OpenCV into rgb8:

$ ck benchmark program:image-classification-tflite --speed --skip_print_timers \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10 \
--dep_add_tags.images=preprocessed,using-opencv,rgb8 \
--dep_add_tags.weights=mobilenet,quantized
...
Summary:
-------------------------------
Graph loaded in 0.001789s
All images loaded in 0.000663s
All images classified in 0.477427s
Average classification time: 0.043434s
Accuracy top 1: 0.7 (7 of 10)
Accuracy top 5: 0.9 (9 of 10)
--------------------------------
psyhtest commented 5 years ago

Perhaps it's somewhat similar to https://github.com/mlperf/inference/issues/334?