search

sign-language-processing / spoken-to-signed-translation

a text-to-gloss-to-pose-to-video pipeline for spoken to signed language translation
https://sign.mt/?sil=sgg&spl=de
MIT License
47 stars 12 forks source link

March 2024 Progress #26

Open AmitMY opened 7 months ago

AmitMY commented 7 months ago

We now use the improved pose-to-video based on diffusion models.

We start with a paragraph in German, translate it to German Sign Language:

Das Alte Museum wurde 1830 als erstes öffentliches Museum in Berlin eröffnet. Im Obergeschoss können Sie bei einem großartigen Ausblick über den Lustgarten später mehr über die Geschichte des Museums und seinen Architekten Karl Friedrich Schinkel erfahren.

The simple glossing gives:

[("das", "der"), ("alte", "alter"), ("museum", "museum"), ("wurde", "werden"), ("1830", "1830"), ("als", "als"), ("erstes", "erster"), ("öffentliches", "öffentlich"), ("museum", "museum"), ("in", "in"), ("berlin", "berlin"), ("eröffnet", "eröffnet"), (".", "."), ("im", "im"), ("obergeschoss", "obergeschoss"), ("können", "können"), ("sie", "sie|sie"), ("bei", "bei"), ("einem", "ein"), ("großartigen", "großartig"), ("ausblick", "ausblick"), ("über", "über"), ("den", "der"), ("lustgarten", "lustgarten"), ("später", "spät"), ("mehr", "mehr"), ("über", "über"), ("die", "der"), ("geschichte", "geschichte"), ("des", "der"), ("museums", "museum"), ("und", "und"), ("seinen", "sein"), ("architekten", "architekt"), ("karl", "karl"), ("friedrich", "friedrich"), ("schinkel", "schinkel"), ("erfahren", "erfahren"), (".", ".")]

The current system gives:

TODO

We choose to focus on one issue - visual inconsistency between signs. After adding pose anonymization in https://github.com/sign-language-processing/spoken-to-signed-translation/commit/0072c52478c5cf9be030b3181bc997337e6507f3 the output is:

https://github.com/sign-language-processing/spoken-to-signed-translation/assets/5757359/12c1dc04-043d-4709-845d-303f3407b2ce

We note that the database lookup time was 9 seconds, this was optimized to 1-2 seconds, and could be improved further.

We recognize that sentences should be split. This will affect both the database search (only search up to one sentence) and in the video, lower and raise hands without cropping on sentence boundary. (Possibly, generate every sentence independently, then join them)

AmitMY commented 1 month ago

Some notes:

  1. The system mainly fails here on numbers (1830), and named entities (Karl Friedrich Schinkel) (which with some modifications, it could spell out).

  2. It also did not make the sentence boundary clear, and basically ignored the punctuation (also, fixable).

  3. In my opinion maybe the biggest problem here to address, is that the signing is performed in the spoken language word order. It is comprehensible, but not really sign language.

  4. The smoothing between signs is too simplistic (can be easily seen in the skeleton video), and can be fixed.

  5. The video quality is not the best. The generated interpreter has some artifacts even if the pose sequence was perfect. Not easily fixable. 

  6. The video is quite slow. Further work can be done to make the signing faster and tighter, decreasing the number of frames.