Enhancing Pipeline with Vietnamese Language Support

[nqkhanh2002]

Description

I want to enhance my existing pipeline, which follows the flow text -> glosses -> pose sequence, to include Vietnamese language support. Currently, Vietnamese language detection is implemented using Google's cld3, but the recognition part is missing. I need guidelines for adding Vietnamese language support, including any necessary data or modifications to the existing pipeline.

Expected Behavior

• The pipeline should support Vietnamese language recognition.
• The flow from text -> glosses -> pose sequence should work seamlessly for Vietnamese.

Actual Behavior

• Vietnamese language recognition is not implemented in the pipeline.

Steps to Reproduce the Problem

• Review the pipeline steps from text -> glosses -> pose sequence.
• Identify the necessary modifications to include Vietnamese language support.
• Implement the modifications.

Environment

• Platform: Windows 10
• Browser: Google Chrome 123
• Additional Information: Vietnamese language detection is implemented using Google's cld3

[nqkhanh2002]

Hi @AmitMY , Could you please provide an update on the status of this issue? Timely feedback would be greatly appreciated, thanks

[AmitMY]

Thank you for opening this issue. To support Vietnamese sign language in the current approach of text->gloss->pose, please see https://github.com/sign-language-processing/spoken-to-signed-translation It will allow you to load a dictionary (in a specific format) such that vietnamese is supported.

Then you will have two options:

1. Expose this tool using an API, and call it from a local version of sign/translate
2. Share the dictionary with me, and I will make it work in https://sign.mt alongside other languages.

[nqkhanh2002]

Hi @AmitMY ,

I have run the repository using the download_lexicon.py file, and as I understand, the output folder will be the dictionary that I can share with you. However, I encountered an issue as the dataset used in the demo repo is signsuisse, which only supports Swiss, French, German, and Italian languages. I have also searched for datasets supporting Vietnamese at sign-language-processing/datasets, but they do not exist. What should be my next steps to proceed with adding Vietnamese language support? Your timely feedback would be greatly appreciated as this is quite urgent. Thank you very much!

[AmitMY]

There seems to be a dictionary available for Vietnamese - https://tudienngonngukyhieu.com/tu-ngu-theo-bang-chu-cai

Your steps should be as follows:

1. Add support for this dictionary in sign-language-processing/datasets (Make a PR)
2. Run pose estimation on the data (video-to-pose -i video.mp4 -o video.pose)
3. Edit download_lexicon to support this dataset from the datasets library

Then, you will have a local demo running with this dictionary, and you will be able to apply some modifications if you want.

[nqkhanh2002]

Hi @AmitMY, I created PR at #pull/73 Please review to see if I did the job correctly, thank you very much

[nqkhanh2002]

Hi @AmitMY I also reviewed possible flows in sign-language-processing/spoken-to-signed-translation such as: text_to_gloss_to_pose , text_to_gloss , pose_to_video but did not see any flows related to video_to_pose as you mentioned. I also reviewed sign-language-processing and saw a lot of work repo/work being created here. Does this project have any other project documentations besides the wiki and project board or am I missing something? Right now I don't know where to start next for the betterment of Vietnamese. I think next I will improve the human liver while waiting for instructions. Thank you very much

[AmitMY]

For sign-language-processing/spoken-to-signed-translation you need to extract poses - similar to the dummy lexicon - https://github.com/sign-language-processing/spoken-to-signed-translation/tree/main/assets/dummy_lexicon

the video_to_pose command exists in the https://github.com/sign-language-processing/pose package (pip install pose-format)