search

sign / translate

Effortless Real-Time Sign Language Translation
https://sign.mt
Other
496 stars 85 forks source link

Enhancing Pipeline with Vietnamese Language Support #159

Closed nqkhanh2002 closed 4 months ago

nqkhanh2002 commented 5 months ago

Description

I want to enhance my existing pipeline, which follows the flow text -> glosses -> pose sequence, to include Vietnamese language support. Currently, Vietnamese language detection is implemented using Google's cld3, but the recognition part is missing. I need guidelines for adding Vietnamese language support, including any necessary data or modifications to the existing pipeline.

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Environment

nqkhanh2002 commented 5 months ago

Hi @AmitMY , Could you please provide an update on the status of this issue? Timely feedback would be greatly appreciated, thanks

AmitMY commented 5 months ago

Thank you for opening this issue. To support Vietnamese sign language in the current approach of text->gloss->pose, please see https://github.com/sign-language-processing/spoken-to-signed-translation It will allow you to load a dictionary (in a specific format) such that vietnamese is supported.

Then you will have two options:

  1. Expose this tool using an API, and call it from a local version of sign/translate
  2. Share the dictionary with me, and I will make it work in https://sign.mt alongside other languages.
nqkhanh2002 commented 5 months ago

Hi @AmitMY ,

I have run the repository using the download_lexicon.py file, and as I understand, the output folder will be the dictionary that I can share with you. However, I encountered an issue as the dataset used in the demo repo is signsuisse, which only supports Swiss, French, German, and Italian languages. I have also searched for datasets supporting Vietnamese at sign-language-processing/datasets, but they do not exist. What should be my next steps to proceed with adding Vietnamese language support? Your timely feedback would be greatly appreciated as this is quite urgent. Thank you very much!

AmitMY commented 5 months ago

There seems to be a dictionary available for Vietnamese - https://tudienngonngukyhieu.com/tu-ngu-theo-bang-chu-cai

Your steps should be as follows:

  1. Add support for this dictionary in sign-language-processing/datasets (Make a PR)
  2. Run pose estimation on the data (video-to-pose -i video.mp4 -o video.pose)
  3. Edit download_lexicon to support this dataset from the datasets library

Then, you will have a local demo running with this dictionary, and you will be able to apply some modifications if you want.

nqkhanh2002 commented 4 months ago

Hi @AmitMY, I created PR at #pull/73 Please review to see if I did the job correctly, thank you very much

nqkhanh2002 commented 4 months ago

Hi @AmitMY I also reviewed possible flows in sign-language-processing/spoken-to-signed-translation such as: text_to_gloss_to_pose , text_to_gloss , pose_to_video but did not see any flows related to video_to_pose as you mentioned. I also reviewed sign-language-processing and saw a lot of work repo/work being created here. Does this project have any other project documentations besides the wiki and project board or am I missing something? Right now I don't know where to start next for the betterment of Vietnamese. I think next I will improve the human liver while waiting for instructions. Thank you very much

AmitMY commented 4 months ago

For sign-language-processing/spoken-to-signed-translation you need to extract poses - similar to the dummy lexicon - https://github.com/sign-language-processing/spoken-to-signed-translation/tree/main/assets/dummy_lexicon

the video_to_pose command exists in the https://github.com/sign-language-processing/pose package (pip install pose-format)