search

sign / translate

Effortless Real-Time Sign Language Translation
https://sign.mt
Other
414 stars 74 forks source link

[Feature] Fingerspelling for Names (nouns) #118

Closed imsamimalik closed 8 months ago

imsamimalik commented 8 months ago

Problem

No finger spelling for names (or in general unknown nouns)

Description

Right now If I try any name it tends to produce the same result every time.

I tried the prompt My name is Abraham. or I put in any other name instead of Abraham it produced the same result. The produced video is attached, My-name-is-Abraham.webm

Alternatives

I think fingerspelling should be used for any nouns (names specifically) that don't have any translation in the dataset.

Additional context

No response

AmitMY commented 8 months ago

Thanks for this issue.

Referencing the pipeline: https://github.com/sign/translate/wiki/Spoken-to-Signed

image

Our directions for this is the following:

  1. fast: text-to-gloss-to-pose - this should be implemented here where by default, when not finding a word it should break it down into characters. The problem is that the lexicon does not include videos of letters as of yet. Another issues is that, for example, this pipeline signs "is" in "My name is" which is not usually signed in signed langauges.
  2. slow: text-to-SignWriting-to-pose - this is being actively worked on, specifically, the text-to-SignWriting is being trained from here. In it's data, there are many name instances, like "My name is Amit" in parallel with the SignWriting translation (see below). Once this is working consistently, and the translation works well, we should move on to SignWriting-to-pose animation, which will take some time to research. image
imsamimalik commented 8 months ago

Thanks for the response. It is very weird that the Lexicon does not contain the basic alphabet.

Another issues is that, for example, this pipeline signs "is" in "My name is" which is not usually signed in signed languages.

I thought the Reordering and Dropping Component in here was doing the exact thing; dropping unnecessary words.

Nonetheless, thanks for your efforts.

AmitMY commented 8 months ago

That is right - the spacy based component does that, but in user testing we found that people prefer it less to the component that doesn't (to our great surprise) So further adjustments are required