search

jina-ai / serve

☁️ Build multimodal AI applications with cloud-native stack
https://jina.ai/serve
Apache License 2.0
21.13k stars 2.22k forks source link

Project Idea : Text/audio to Sign Language Video converter #4462

Closed BabyElias closed 2 years ago

BabyElias commented 2 years ago

Backdrop:

Around 466 million people worldwide currently have a hearing loss and 34 million of those individuals are children. Hearing loss is increasing every year, and it is estimated that by 2050, over 900 million people will have a hearing loss. Sign language is a natural way of communication for challenged people with speaking and hearing disabilities. There have been various mediums available to translate or to recognize sign language and convert them to text, but text/audio to sign language conversion systems have been rarely developed due to the scarcity of any sign language corpus.

Problems Identified : 1) Such interactive systems are already developed for sign language systems like ASL and BSL but most of them generate only images representing the text which is not as realistic as video output. 2) Also,these interactive systems are based on word to word conversion. The meaning and context of sentences are not taken in consideration which might result in undesired output. 3) A number of sign languages such as ISL don't even have these basic interactive systems developed for them ,even though 1.8 million people of the country use it.

Proposed Solution :

A system where we can input a speech in text/audio format and it will generate a video that converts the speech/text to sign language.

The text conversion system can work on the following algorithm- a. English parser for parsing the English text b. Sentences reordering module based on the corresponding sign language grammar rules c. tokenizer which splits the text into words d. Eliminator for eliminating stopwords e. Stemming for getting the root words of each word and synonym replacement for words not in dictionary. f. Video conversion module.

These algorithms take inspiration from - https://www.cse.scu.edu/~m1wang/projects/NLP_English2IndianSignLanuguage_18w.pdf
https://www.interscience.in/cgi/viewcontent.cgi?article=1170&context=ijcsi (No plagiarism intended)

Abbhiishek commented 2 years ago

Hey @BabyElias i am a mentee of OpenForce2022 . I would like to work on it . I don't have any prior knowledge regarding the tech stacks but i want to give it a try just for fun and opensource contributions.

Can you mentor me ?

Looking Forward to your Reply !

BabyElias commented 2 years ago

Hello @Abbhiishek ! I too actually am a newbie in this field.I came up with this idea in one of the brainstorming sessions we had as part of an Open-Source mentorship cohort. After going through Jina's learning portals, I conceptualized this idea as a contribution to Jina. So if you want to seek mentorship regarding this project, I would say it would be best to contact @joeyouss Mam for the concerns.

Thanks for your interest & All the best for working on it!

jina-bot commented 2 years ago

@jina-ai/product This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 14 days