Measure performance and accuracy on customized dataset

mlcommons / tiny

MLPerf™ Tiny is an ML benchmark suite for extremely low-power systems such as microcontrollers

https://mlcommons.org/en/groups/inference-tiny/

Apache License 2.0

344 stars 81 forks source link

Measure performance and accuracy on customized dataset #109

Open BeyondCloud opened 3 years ago

BeyondCloud commented 3 years ago

Hi, Is there any way to measure performance and accuracy on customized dataset (other than ad, vww, kws, ic) ? I have tried setting EE_MODEL_VERSION to my own dataset and put the dataset under /ulp-mlperf/datasets/ directory, but I got the following error when running medium performance:

Need at least 5 inputs to run benchmark mode