cleong110
commented
4 months ago Should maybe also add the paper associated with OpenHands, and talk about their "sign2vec" approach?
edit: it's in there already, good.
Selvaraj et al. (2022) introduced an open-source OpenHands library, which consists of standardized pose datasets for different existing sign language datasets and trained checkpoints of four pose-based isolated sign language recognition models across six languages (American, Argentinian, Chinese, Greek, Indian, and Turkish). To address the lack of labeled data, they propose self-supervised pretraining on unlabeled data and curate the largest pose-based pretraining dataset on Indian Sign Language (Indian-SL). They established that pretraining is effective for sign language recognition by demonstrating improved fine-tuning performance especially in low-resource settings and high crosslingual transfer from Indian-SL to a few other sign languages.
The work of Kezar, Thomason, and Sehyr (2023), based on the OpenHands library, explicitly recognizes the role of phonology to achieve more accurate isolated sign language recognition (ISLR). To allow additional predictions on phonological characteristics (such as handshape), they combine the phonological annotations in ASL-LEX 2.0 (Sehyr et al. 2021) with signs in the WLASL 2000 ISLR benchmark (Li et al. 2020). Interestingly, Tavella et al. (2022) construct a similar dataset aiming just for phonological property recognition in American Sign Language (ASL).
Edit 2: seems I already investigated this, whoops. https://cdleong.github.io/slt-guide/#openhands-possibly-a-dead-project
https://openhands.ai4bharat.org/en/latest/index.html
Potentially useful for:
Currently it seems a bit unstable though. pip install + import does not work in Colab.