search

cedrickchee / pytorch-android

[EXPERIMENTAL] Demo of using PyTorch 1.0 inside an Android app. Test with your own deep neural network such as ResNet18/SqueezeNet/MobileNet v2 and a phone camera.
Other
106 stars 18 forks source link
android-application caffe2 caffe2pytorch deep-learning deployment onnx pytorch-cnn resnet18 squeezenet

readme

Deprecation Notice

As of this announcement PyTorch 1.3 now officialy supports an end-to-end workflow from Python to deployment on iOS and Android through PyTorch Mobile. Thank you for your interest with this project.

PyTorch on Android

PyTorch on Android is a project to demo how to use PyTorch and ONNX to build an Android mobile application doing real time object classification.

demo

The source code for the demo in this repo was originally based on AICamera repo and as of 2018-12-31, the codebase was based on Soumith's AICamera repo.

Project Status:

Goal

Make it easier to ship and test your neural network model in PyTorch on mobile devices.

Documentation

Updating the AICamera Android app to work with Caffe2 from PyTorch/master

This is an example for using Caffe2 on Android.

  1. git clone PyTorch source and switch to remote branch, android_oss_fixes:
$ git clone --recursive https://github.com/pytorch/pytorch.git
$ cd pytorch
# We are using this PyTorch master commit 39381964bbb2686cf84c78aed638985455f6d3fe (Dec 23 02:08:33 2018 -0500, "fix build_android for newest NDK, and make install work")
$ git checkout --track origin/android_oss_fixes

M third_party/QNNPACK
M third_party/cpuinfo
M third_party/fbgemm
M third_party/ideep
Branch 'android_oss_fixes' set up to track remote branch 'android_oss_fixes' from 'origin'.
Switched to a new branch 'android_oss_fixes'

  1. You'll need to download the Android NDK (latest r18 release) if you have not.

  2. First, we will compile Caffe2 Android libraries like libcaffe2, libqnnpack for arm-v7a ABI and then for x86.

Set build environment:

# make sure $PYTORCH_ROOT, $AICAMERA_ROOT and $ANDROID_NDK are set
export ANDROID_NDK=~/android/sdk/ndk-bundle/
export PYTORCH_ROOT=~/dev/gh/pytorch/
export AICAMERA_ROOT=~/dev/android/android-studio-projects/aicamera/

pushd $PYTORCH_ROOT

Then, do the following:

Build Caffe2 android libs and copy them over into AICamera app folder

./scripts/build_android.sh

If you encountered build errors related to "quantized/int8_*.cc", get the patch from this "Update QNNPACK" PR. Patch these files in your local copies.

You should see "Install configuration: "Release"" output in your terminal if the process completed successfully.

mv build_android build_android_arm

# copy headers
cp -r install/include/* $AICAMERA_ROOT/app/src/main/cpp/

# copy arm libs
rm -rf $AICAMERA_ROOT/app/src/main/jniLibs/armeabi-v7a/
mkdir $AICAMERA_ROOT/app/src/main/jniLibs/armeabi-v7a
cp -r build_android_arm/lib/lib* $AICAMERA_ROOT/app/src/main/jniLibs/armeabi-v7a/

./scripts/build_android.sh -DANDROID_ABI=x86
mv build_android build_android_x86

# copy x86 libs
rm -rf $AICAMERA_ROOT/app/src/main/jniLibs/x86/
mkdir $AICAMERA_ROOT/app/src/main/jniLibs/x86
cp -r build_android_x86/lib/lib* $AICAMERA_ROOT/app/src/main/jniLibs/x86/
  1. Build the AICamera app using the Build -> Make Project menu option in Android Studio

If you prefer not to build PyTorch from master, I have commited the files for the Caffe2 bits into this git repo. Unfortunately, not all files are uploaded to GitHub due to file size limit.

Grab the remaining large files from my Google Drive.

Developer Guide

We created a developer guide on how to ship a convolutional neural network (Resnet18 and SqueezeNet) on Android with PyTorch and Android Studio. Check it out!

Step by Step Guide

We'll walk you through every step, from problem all the way to building and deploying the Android app to mobile phones.

Features, Functionalities and TODO

Android Project

You can download the Android project source code by running this command:

git clone https://github.com/cedrickchee/pytorch-android.git

Android Development Environment

Dependencies

Tests

Device Network FPS (^)
Google Nexus 6P SqueezeNet 3.0
Google Nexus 6P Resnet18 0.6
Samsung Note 8 SqueezeNet TBD
Galaxy Note 3 SqueezeNet 4.0
Samsung Galaxy S7 SqueezeNet 5.8
Google Pixel SqueezeNet 5.7

Note: ^ the number of FPS is subjective to the camera photo (image) size you send to the mobile device as well as type of the device.

Demo

If building this repo is too much of a trouble for you, we also plan to put this in Google Play Store (TBD).

Network Graph Comparison

Visualize SqueezeNet 1.1 prediction network serialized as Protobuf file:

Before update (Oct 2018) After update (Jan 2019)

Performance

TBD

Future Work

Scope of work for this project: TBD

Other Useful Resources

PyTorch official forums

GitHub Issues and Commits

Fast.ai

Fast Neural Style Transfer (PyTorch → ONNX→ CoreML/NNAPI)

Support

Feel free to ask any questions, from preparing development environment to debugging on Android Studio. We are happy to help you.

License

This repository contains a variety of content; some developed by Cedric Chee, and some from third-parties. The third-party content is distributed under the license provided by those parties.

I am providing code and resources in this repository to you under an open source license. Because this is my personal repository, the license you receive to my code and resources is from me and not my employer.

The content developed by Cedric Chee is distributed under the following license:

Code

The code in this repository, including all code samples in the notebooks listed above, is released under the MIT license. Read more at the Open Source Initiative.

Text

The text content of the book is released under the CC-BY-NC-ND license. Read more at Creative Commons.