search

mlcommons / mobile_models

MLPerf™ Mobile models
https://mlcommons.org/en/groups/inference-mobile/
Apache License 2.0
24 stars 10 forks source link

add snusr dataset in png #62

Closed freedomtan closed 1 year ago

freedomtan commented 1 year ago

for https://github.com/mlcommons/mobile_app_open/issues/726

github-actions[bot] commented 1 year ago

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

freedomtan commented 1 year ago

@freedomtan In the related PR mlcommons/mobile_app_open#742 to use this new dataset, the accuracy test for iOS and Windows failed because the accuracy is only ~0.06 (expected is ~0.33). Can you please check it?

Interesting. How about accuracy number on Android devices? if there is no such problem on Android devices, mostly it's some kind of file reading problem.

anhappdev commented 1 year ago

Interesting. How about accuracy number on Android devices? if there is no such problem on Android devices, mostly it's some kind of file reading problem.

On Android, the accuracy test passed.

freedomtan commented 1 year ago

Interesting. How about accuracy number on Android devices? if there is no such problem on Android devices, mostly it's some kind of file reading problem.

On Android, the accuracy test passed.

I figured out what the problem is. My first guess was wrong. File reading/permission looks fine. The problem is that we didn't handle floating output correctly in SNUSR::ProcessOutput. I screwed up my MacBook Pro M1 by installing macOS Sonoma on it. With that, I cannot use Xcode 14.x (Xcode 15.x beta failed the app). So I guess I won't be able to work on it this week. If you have time to work on it, you can take it.

Quick test to confirm what the problem: replacing the sr model with a quant one when building the windows tflite app.

freedomtan commented 1 year ago

Interesting. How about accuracy number on Android devices? if there is no such problem on Android devices, mostly it's some kind of file reading problem.

On Android, the accuracy test passed.

it turns out when converting PNG byte array to floating point array, I forgot to add unsigned modifier before char. In case you don't know it in C/C++ the signedness of char is implementation depend. On x86 systems, mostly char means signed char. On most ARM systems except Apple idevices, char means unsigned char.

With https://github.com/mlcommons/mobile_app_open/pull/742/commits/41249354533e011a9772184dfb690897d09dd1b7, this accuracy problem on iOS and Windows is fixed.

anhappdev commented 1 year ago

it turns out when converting PNG byte array to floating point array, I forgot to add unsigned modifier before char. In case you don't know it in C/C++ the signedness of char is implementation depend. On x86 systems, mostly char means signed char. On most ARM systems except Apple idevices, char means unsigned char.

With mlcommons/mobile_app_open@4124935, this accuracy problem on iOS and Windows is fixed.

Thank you for the fix and the info.