American Sign Language (ASL) is a natural language used by deaf communities in many countries around the world. It has 26 signs corresponding to the 26 letters of the English language. This repo shows Python scripts that train a deep learning model to recognize the 26 ASL signs (and 3 additional signs for deletion, space, and nothing) and converts and optimizes the model to the Mobile Interpreter format, and an Android app that uses the model to recognize the 26 ASL signs.
To Test Run the ASL recognition Android App, follow the steps below:
1. Train and Prepare the Model
If you don't have the PyTorch 1.9.0 and torchvision 0.10.0 installed, or if don't want to install them, you can skip this step. The trained, scripted and optimized model is already included in the repo, located at ASLRecognitionapp/src/main/assets.
Otherwise, open a terminal window, make sure you have torch 1.9.0 and torchvision 0.10.0 installed using command like pip list|grep torch, or install them using command like pip install torch torchvision, then run the following commands:
git clone https://github.com/pytorch/android-demo-app
cd android-demo-app/ASLRecognition/scripts
Download the ASL alphabet dataset here and unzip it into the ASLRecognition/scripts folder. Then run the scripts below, which are based on this tutorial, to pre-process the training images, train the model and convert and optimize the trained model to the mobile interpreter model:
python preprocess_image.py
python create_csv.py
python train.py --epochs 5 # on a machine without GPU this can take hours
python convert_lite.py
If all goes well, the model asl.ptl will be generated and you can copy it to ASLRecognition/app/src/main/assets.
You can also run python test.py to see the result of a test image located at ../app/src/main/assets/C1.jpg:
Predicted output: C
0.043 seconds
For more information on how to use a test script like the above to find out the expected model input and output and use them in an Android app, see Step 2 of the tutorial Image Segmentation DeepLabV3 on Android.
2. Use Android Studio
Open the ASLRecognition project using Android Studio. Note the app's build.gradle file has the following lines:
Select an Android emulator or device and build and run the app. Some of the 26 test images of the ASL alphabet and their recognition results are as follows:
To test the live ASL alphabet gesture recognition, after you get familiar with the 26 ASL signs by tapping Next and Recognize, select the LIVE button and make some ASL gesture in front of the camera. A screencast of the app running is available here.
American Sign Language Recognition on Android
Introduction
American Sign Language (ASL) is a natural language used by deaf communities in many countries around the world. It has 26 signs corresponding to the 26 letters of the English language. This repo shows Python scripts that train a deep learning model to recognize the 26 ASL signs (and 3 additional signs for deletion, space, and nothing) and converts and optimizes the model to the Mobile Interpreter format, and an Android app that uses the model to recognize the 26 ASL signs.
Prerequisites
Quick Start
To Test Run the ASL recognition Android App, follow the steps below:
1. Train and Prepare the Model
If you don't have the PyTorch 1.9.0 and torchvision 0.10.0 installed, or if don't want to install them, you can skip this step. The trained, scripted and optimized model is already included in the repo, located at
ASLRecognitionapp/src/main/assets.Otherwise, open a terminal window, make sure you have torch 1.9.0 and torchvision 0.10.0 installed using command like
pip list|grep torch, or install them using command likepip install torch torchvision, then run the following commands:Download the ASL alphabet dataset here and unzip it into the
ASLRecognition/scriptsfolder. Then run the scripts below, which are based on this tutorial, to pre-process the training images, train the model and convert and optimize the trained model to the mobile interpreter model:If all goes well, the model
asl.ptlwill be generated and you can copy it toASLRecognition/app/src/main/assets.You can also run
python test.pyto see the result of a test image located at../app/src/main/assets/C1.jpg:For more information on how to use a test script like the above to find out the expected model input and output and use them in an Android app, see Step 2 of the tutorial Image Segmentation DeepLabV3 on Android.
2. Use Android Studio
Open the ASLRecognition project using Android Studio. Note the app's
build.gradlefile has the following lines:and in the MainActivity.java, the code below is used to load the model:
3. Run the App
Select an Android emulator or device and build and run the app. Some of the 26 test images of the ASL alphabet and their recognition results are as follows:
To test the live ASL alphabet gesture recognition, after you get familiar with the 26 ASL signs by tapping Next and Recognize, select the LIVE button and make some ASL gesture in front of the camera. A screencast of the app running is available here.
4. What's Next
With a different sign language dataset such as the RWTH-PHOENIX-Weather 2014 MS Public Hand Shape Dataset or the Continuous Sign Language Recognition Dataset and a state-of-the-art sign language transformer based model, more powerful sign language recognition Android app can be developed based on the app here.