search

mlcommons / mobile_models

MLPerf™ Mobile models
https://mlcommons.org/en/groups/inference-mobile/
Apache License 2.0
24 stars 10 forks source link

Cannot reproduce Qualcomm results with released model on Samsung S21 Ultra (Snapdragon version) #22

Closed swb1234554321 closed 3 years ago

swb1234554321 commented 3 years ago

Hi,

When trying to reproduce mlperf-mobilev1.0 results submitted by Qualcomm, we found our results is much worse than the claimed ones, especially for single-stream mode.

Environment settings:

Device: both on HDK and Samsung S21 pro (got very close score).
SDK version: 1.48.0.2554. Model files: Tried all of the available '.dlc' files under this repo. Scripts used for benchmarking: snpe_bench.py included in SNPE v1.48.0.2554

Results comparison:

image

Is there any released code implementation to reproduce the submitted results? Based on the current public dlc files and SNPE SDK, there seem to be 30%~60% gaps between measured & submitted FPS score.

Thanks.

mcharleb commented 3 years ago

Is there any released code implementation to reproduce the submitted results?

The code for QTI backend was released as part of the 1.0 submission. The results were verified on a OnePlus 9 Pro and a Realme 5G

mcharleb commented 3 years ago

Running snpe_bench.py is not equivalent to running the MLPerf app.

mcharleb commented 3 years ago

I do not have S21 Ultra to try the MLPerf app on

swb1234554321 commented 3 years ago

Thanks for replying. Could you please provide a public link to the code or MLPerf mobile app? That will help a lot.

swb1234554321 commented 3 years ago

Running snpe_bench.py is not equivalent to running the MLPerf app.

What is the difference between snpe_bench.py and MLPerf app measurement? DSP runtime libs, measurement methodology or something?

mcharleb commented 3 years ago

Mobile app code is now public: https://github.com/mlcommons/mobile_app_open