handshape

Goal

There are various handshape datasets for Sign Language. However:

Each dataset has its own format and many are hard to find.
Each dataset has its own mapping of handshapes to classes. While signs depend on the specific Sign Language for a country/region, handshapes are universal. Hence, they could be shared between datasets/tasks.

This library aims to provide two main features:

A simplified API to download and load handshape datasets
A mapping between datasets so that datasets can be merged for training/testing models.

This library is a work in progress. Contributions are welcome. If you wish to add a dataset you can make a push request or open an issue.

Installation

You can install handshape_datasets via pip with:

pip install handshape_datasets

Basic usage

Simply import the module and load a dataset. The following downloads, preprocesses and load to memory the LSA16 dataset:

import handshape_datasets as hd
images,metadata = hd.load("lsa16")

Afterwards you can, for example, plot the first images of the dataset

import matplotlib.pyplot as plt
plt.imshow(images[0,:,:,:]) # show the first sample of the dataset

Advanced usage

import handshape_datasets as hd
hd.list_datasets() # List available datasets
hd.load("lsa16",version="color",delete=True) # use the color version, delete temporary files
hd.delete_temporary_files("lsa16")# Delete temporary files  (if any)
hd.clear("lsa16") # Delete all the local files for dataset LSA16
hd.info("lsa16") # Shows detailed info of the dataset, including url, data format, fields, etc.

Supported datasets

Dataset id	Download size	Size on disk	Samples	Classes
lsa16	640.6 Kb	1.2 Mb	800	16
rwth	44.8 Mb	206.8 Mb	3359	45
Irish	173.4 Mb	515.0 Mb	58114	26
Ciarp	10.6 Mb	18.6 Mb	7127	10
PugeaultASL_A	2.1 Gb	4.3 Gb	65774	24
PugeaultASL_B	317.4 Mb	717.9 Mb	72676	26
indianA	1.7 Gb	1.9 Gb	5040	140
indianB	320.5 Mb	8.6 Gb	5000	140
Nus1	2.8 Mb	3.6 Mb	479	10
Nus2	73.7 Mb	106.1 Mb	2750	10
jsl	4.5 Mb	7.9 Mb	8055	41
psl	285.2 Mb	1.2 Gb	960	16

You can find more information about the datasets in the following sign language dataset survey

Training a handshape classifier with Keras

Load the dataset:

x,metadata = handshape_datasets.load("lsa16")
y = metadata["y"]

Get the input_shape and number of classes:

input_shape = x[0].shape
classes = y.max() + 1

Define a model (using a pretrained MobileNet here):

base_model = keras.applications.mobilenet.MobileNet(input_shape=(input_shape[0],input_shape[1],3), 
                                                            weights='imagenet', include_top=False)
output = keras.layers.GlobalAveragePooling2D()(base_model.output)
output = keras.layers.Dense(32, activation='relu')(output)
output = keras.layers.Dense(classes, activation='softmax')(output)
model = Model(inputs=base_model.input, outputs=output)
model.compile(optimizer='Adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

Split the dataset intro train/test sets:

X_train, X_test, Y_train, Y_test = sklearn.model_selection.train_test_split(x,y,
                                                                                test_size=0.9,
                                                                                stratify=y)

Fit the model

history = model.fit(X_train, Y_train, batch_size=self.batch_size, epochs=self.epochs, validation_data=(X_test, Y_test))

Google Colab example:

https://colab.research.google.com/drive/1kY-YrbegGFVT7NqVaeA4RjXYRVlZiISR?usp=sharing

midusi / handshape_datasets