IPA to SignWriting

[AmitMY]

For the creation of a lip-reading dataset in SignWriting, we need to map IPA symbols to SignWriting.

The project will go like this:

1. Collect sign language videos with a known language (e.g. English) and a word / phrase
2. Check if the word was mouthed during signing (perhaps use models from https://www.robots.ox.ac.uk/~vgg/data/lip_reading/)
3. If so, translate from the spoken language to IPA (can be erroneous, people vary in pronunciation)
4. Automatically generate the mouthing in SignWriting, for model training

https://www.signwriting.org/archive/docs7/sw0617_Cross_Linguistic_Guide_SignWriting_Parkhurst.pdf

Mapping: https://www.signwriting.org/archive/docs2/sw0124-DE-Mundbildschrift.pdf

[AmitMY]

Started here https://github.com/sign-language-processing/signwriting/tree/main/signwriting/mouthing

[AmitMY]

Good datasets to process that are not currently available:

• https://mmai.io/datasets/lip_reading/
• https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrs3-lang.html

Good dataset in German that is available:

• https://www.kaggle.com/datasets/geraldschwiebert/glips-german-lipreading-dataset
  • need to merge m4a and mp4 files
  • need to find the exact timestamp of the word pronunciation
  • will need to only run face model, and upload pose files, before uploading videos (scale and shift)

[AmitMY]

Done!