Open Mostelk opened 6 months ago
Should this feature be in the app?
On a computer, it's quite easy to zip and split the big file, though. For example, on macOS I can zip and split the file with this command:
zip -s 25m new.zip big_file.json
But doing that in the mobile app is not trivial.
@anhappdev it is nice to have, not a must, I know split can do it. There is some reference here as well https://stackoverflow.com/questions/37761543/how-i-can-split-file-in-android-case-the-file-is-large-to-upload-itsound-file
Should this feature be in the app?
On a computer, it's quite easy to zip and split the big file, though. For example, on macOS I can zip and split the file with this command:
zip -s 25m new.zip big_file.json
But doing that in the mobile app is not trivial.
I don't remember the status of zip, but I think the old unix command split
is available on Android.
I don't remember the status of zip, but I think the old unix command split is available on Android.
But the solution must be cross-platform and work on Android, iOS and Windows.
generally, use some libz (zip, gzip, or compression form) or any libraries we can use to do what the old unix split
does. @freedomtan to dig more
or we can just add it to loadgen. Do the splitting during log generation time?
Need a post processing script in the App, that checks the size of the mlperf_log_trace, and splits it into smaller sizes for upload results to github, for example https://github.com/mlcommons/mobile_results_v3.1/tree/main/closed/Qualcomm_Innovation_Center/results/SD7PG2/mobilenetEdgeTPU/Offline/log_accuracy splits the json files to chunks of 25Mb https://github.com/mlcommons/mobile_results_v3.1/tree/main/closed/Samsung/results/object_detection-performance/mlperf_log_trace_split splits them into chunks < 50 MB
need to apply to the v4 submission branch https://github.com/mlcommons/mobile_app_open/tree/submission-v4.0