search

XiaoMi / mobile-ai-bench

Benchmarking Neural Network Inference on Mobile Devices
Apache License 2.0
353 stars 57 forks source link

dry run mobile-ai-bench but fail on snapdragon sdm660, sdm845 #6

Closed PennySHE closed 6 years ago

PennySHE commented 6 years ago

~/xiaomi/mobile-ai-bench$ python tools/benchmark.py --output_dir=output --frameworks=MACE --runtimes=CPU --model_names=MobileNetV1 --target_abis=armeabi-v7a Prepare to run models on armeabi-v7a

===================================================================== Trying to lock device d699ac95 Run on device: d699ac95, sdm660, sdm660 for arm64 Adjust power to performance mode on d699ac95, sdm660 adbd is already running as root remount succeeded /system/bin/sh: can't create /sys/class/devfreq/1d84000.ufshc/governor: No such file or directory /system/bin/sh: can't create /sys/class/devfreq/aa00000.qcom,vidc:arm9_bus_ddr/governor: No such file or directory /system/bin/sh: can't create /sys/class/devfreq/aa00000.qcom,vidc:bus_cnoc/governor: No such file or directory /system/bin/sh: can't create /sys/class/devfreq/aa00000.qcom,vidc:venus_bus_ddr/governor: No such file or directory /system/bin/sh: can't create /sys/class/devfreq/aa00000.qcom,vidc:venus_bus_llcc/governor: No such file or directory /system/bin/sh: can't create /sys/class/devfreq/soc:qcom,l3-cdsp/governor: No such file or directory /system/bin/sh: can't create /sys/class/devfreq/soc:qcom,l3-cpu0/governor: No such file or directory /system/bin/sh: can't create /sys/class/devfreq/soc:qcom,l3-cpu4/governor: No such file or directory /system/bin/sh: can't create /sys/class/devfreq/soc:qcom,llccbw/governor: No such file or directory /system/bin/sh: can't create /sys/class/devfreq/soc:qcom,snoc_cnoc_keepalive/governor: No such file or directory 647000000 /system/bin/sh: can't create /d/dri/0/debug/core_perf/perf_mode: No such file or directory /system/bin/sh: can't create /sys/block/sda/queue/nr_requests: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu0/pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu0/rail-pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu1/pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu1/rail-pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu2/pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu2/rail-pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu3/pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu3/rail-pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu4/pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu4/rail-pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu5/pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu5/rail-pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu6/pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu6/rail-pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu7/pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu7/rail-pc/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/l3-wfi/idle_enabled: No such file or directory /system/bin/sh: can't create /sys/module/lpm_levels/L3/llcc-off/idle_enabled: No such file or directory Traceback (most recent call last): File "tools/benchmark.py", line 225, in main(unused_args=[sys.argv[0]] + unparsed) File "tools/benchmark.py", line 217, in main frameworks, model_names, runtimes, output_dir=FLAGS.output_dir) File "/home/xiaomi/mobile-ai-bench/tools/sh_commands.py", line 333, in adb_run prepare_device_env(serialno, abi, device_bin_path, frameworks) File "/home/xiaomi/mobile-ai-bench/tools/sh_commands.py", line 234, in prepare_device_env device_bin_path, serialno) File "/home/xiaomi/mobile-ai-bench/tools/sh_commands.py", line 121, in adb_push adb_push_file(src_path, dst_dir, serialno) File "/home/xiaomi/mobile-ai-bench/tools/sh_commands.py", line 107, in adb_push_file _out=lambda line: stdout_buff.append(line)) File "/usr/local/lib/python2.7/dist-packages/sh.py", line 1427, in call return RunningCommand(cmd, call_args, stdin, stdout, stderr) File "/usr/local/lib/python2.7/dist-packages/sh.py", line 774, in init self.wait() File "/usr/local/lib/python2.7/dist-packages/sh.py", line 792, in wait self.handle_command_exit_code(exit_code) File "/usr/local/lib/python2.7/dist-packages/sh.py", line 815, in handle_command_exit_code raise exc sh.ErrorReturnCode_1:

RAN: /bin/adb -s d699ac95 shell md5sum /data/local/tmp/aibench/libhexagon_controller.so

STDOUT:

STDERR: md5sum: /data/local/tmp/aibench/libhexagon_controller.so: No such file or directory

PennySHE commented 6 years ago

----------------------------------sdm845---------------------------------- ~/xiaomi/mobile-ai-bench$ python tools/benchmark.py --output_dir=output --frameworks=MACE --runtimes=CPU --model_names=MobileNetV1 --target_abis=armeabi-v7a Prepare to run models on armeabi-v7a

===================================================================== Trying to lock device a268e3ca Run on device: a268e3ca, sdm845, SDM845 for arm64 Adjust power to performance mode on a268e3ca, sdm845 adbd is already running as root remount succeeded 710000000 Traceback (most recent call last): File "tools/benchmark.py", line 225, in main(unused_args=[sys.argv[0]] + unparsed) File "tools/benchmark.py", line 217, in main frameworks, model_names, runtimes, output_dir=FLAGS.output_dir) File "/home/xiaomi/mobile-ai-bench/tools/sh_commands.py", line 333, in adb_run prepare_device_env(serialno, abi, device_bin_path, frameworks) File "/home/xiaomi/mobile-ai-bench/tools/sh_commands.py", line 234, in prepare_device_env device_bin_path, serialno) File "/home/xiaomi/mobile-ai-bench/tools/sh_commands.py", line 121, in adb_push adb_push_file(src_path, dst_dir, serialno) File "/home/xiaomi/mobile-ai-bench/tools/sh_commands.py", line 107, in adb_push_file _out=lambda line: stdout_buff.append(line)) File "/usr/local/lib/python2.7/dist-packages/sh.py", line 1427, in call return RunningCommand(cmd, call_args, stdin, stdout, stderr) File "/usr/local/lib/python2.7/dist-packages/sh.py", line 774, in init self.wait() File "/usr/local/lib/python2.7/dist-packages/sh.py", line 792, in wait self.handle_command_exit_code(exit_code) File "/usr/local/lib/python2.7/dist-packages/sh.py", line 815, in handle_command_exit_code raise exc sh.ErrorReturnCode_1:

RAN: /bin/adb -s a268e3ca shell md5sum /data/local/tmp/aibench/libhexagon_controller.so

STDOUT:

STDERR: md5sum: /data/local/tmp/aibench/libhexagon_controller.so: No such file or directory

lee-bin commented 6 years ago

What's your ADB and python sh module version? Is it the same with Environment requirement ? In my working environment, It does not throw an exception while executing md5sum if the target path does not exist.

PennySHE commented 6 years ago

@lee-bin my setup goes well on MACE. My failure on mobile-ai-bech is not MD5 failure, it's caused by lots of adb command exe failed on "no such file or directory"

lee-bin commented 6 years ago

As for logs like

/system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu7/pc/idle_enabled: No such file or directory
/system/bin/sh: can't create /sys/module/lpm_levels/L3/cpu7/rail-pc/idle_enabled: No such file or directory
/system/bin/sh: can't create /sys/module/lpm_levels/L3/l3-wfi/idle_enabled: No such file or directory
/system/bin/sh: can't create /sys/module/lpm_levels/L3/llcc-off/idle_enabled: No such file or directory

, if it fails, the program will just ignore it, it's just for rooted phone.

As for the actual reason, let me explain it in a more explicit way.

  1. Before actual benchmarking, all the libraries, models and data files should be pushed from host to the target phone.
  2. Before actual pushing, the program will check md5sum of the target path and source path, and if it matches, no pushing will be done, so that to speed up the preparing stage. The Python code uses sh module's ADB command to execute the md5sum command.

In my environment, if the target path does not exist, the ADB command in Python code does not throw an exception, just print "No such file or directory". The MACE project does not check md5sum with ADB command before pushing(there are not many files to push, so it's tolerable to always push). You can check the code here.

PennySHE commented 6 years ago

@lee-bin , thanks for your response Ubuntu 16.04 LTS android-ndk-r16b Android Debug Bridge version 1.0.40 gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609 Python 2.7.12 bazel version Build label: 0.13.1

Spend some time to check details it seems like the adb push source file path is not parsed correctly, which cause all required materials are NOT pushed to device.

  1. Prepare device , it will push "libhexagon_controller.so" and do md5sum
  2. Prepare netowrk and input, it will puch model file&mage npy to device and do md5sum

My test fails on #1 bcos the source file path is not passed properly. adb_push("third_party/mace/lib/armeabi-v7a/cpu_gpu_dsp/libhexagon_controller.so", device_bin_path, serialno) adb_push_file(src_path, dst_dir, serialno) home/xiaomi/mobile-ai-bench/tools/sh_commands.py", line 107, in adb_push_file _out=lambda line: stdout_buff.append(line)) File "/usr/local/lib/python2.7/dist-packages/sh.py", line 1427, in call return RunningCommand(cmd, call_args, stdin, stdout, stderr) File "/usr/local/lib/python2.7/dist-packages/sh.py", line 774, in init self.wait() File "/usr/local/lib/python2.7/dist-packages/sh.py", line 792, in wait self.handle_command_exit_code(exit_code) File "/usr/local/lib/python2.7/dist-packages/sh.py", line 815, in handle_command_exit_code raise exc sh.ErrorReturnCode_1:

lee-bin commented 6 years ago

What's your python sh module version? My ADB version is 1.0.32 I don't see any problem with the source file path. The exception is from line 107, which is after checking the source file and is checking the destination file. Anyway, I will handle the exception throwing problem.

PennySHE commented 6 years ago

/usr/local/lib/python2.7/dist-packages/sh.py version = "1.12.14" __project_url__ = "https://github.com/amoffat/sh"

lee-bin commented 6 years ago

It turns out ADB 1.0.40 return exit code 1 after running:

adb shell md5sum /path/to/filename_in_target

, which will cause Python sh module to raise an exception. On the contrary, ADB 1.0.32 return exit code 0 and does not raise an exception. You can check out the latest code which handles both of these situations.

PennySHE commented 6 years ago

Thanks, my workstation is able to run aibench test now