search

eembc / mlmark

EEMBC's Machine-Learning Inference Benchmark targeted at edge devices.
https://www.eembc.org/mlmark
Other
45 stars 5 forks source link

Any reason ResNet50 is different from MLPerf? #1

Closed psyhtest closed 5 years ago

psyhtest commented 5 years ago

The MobileNet and SSD-MobileNet TF models are exactly the same as used in MLPerf, which is great for vendors who might wish to submit to both MLPerf and MLMark. But ResNet50 is different. I think MLMark uses ResNet50-v1.0, while MLPerf uses ResNet50-v1.5. Of course, MLMark is free to choose its own tasks, models and reference implementations. But would not this lead to problems for vendors who wish to support both initiatives?

petertorelli commented 5 years ago

Short answer: it is a coincidence that two of the three models were identical between MLMark and MLPerf. MLMark chose In the future there will be more divergence as more models are added. Ideally this would be a matter for the SDK optimizer, and not per-graph optimizations (analogous to compiler output vs. hand written assembly).