My informal tl;dr would be: Multilingual LLMs are really good at low-resource MT. Can we reformat SL videos into something more compatible with them? Well if we use Vector Quantization to tokenize to "character"-level tokens, then group those tokens into "words" with Optimal Transport, then we have a word-level token "sentence" that LLMs know how to handle. And then you cross-lingual align using MMD to get the tokens to be even more semantically similar. Then you just give it to the LLM and prompt it to translate. And it seems to work well! Without glosses!
Here are my notes on this process. I'm reasonably happy with the summary in terms of accuracy and conciseness, I'm now thinking of how to add value on top of this. Beyond just re-wording and re-summarizing, what would I want to see if I was reading a lit review? Probably a higher-level view, connections with other papers, etc.?
Recent paper with good results, one of the ones from "Awesome" list. The paper's results had not yet been added to https://paperswithcode.com/sota/gloss-free-sign-language-translation-on-csl so I took the liberty of doing that.
My informal tl;dr would be: Multilingual LLMs are really good at low-resource MT. Can we reformat SL videos into something more compatible with them? Well if we use Vector Quantization to tokenize to "character"-level tokens, then group those tokens into "words" with Optimal Transport, then we have a word-level token "sentence" that LLMs know how to handle. And then you cross-lingual align using MMD to get the tokens to be even more semantically similar. Then you just give it to the LLM and prompt it to translate. And it seems to work well! Without glosses!
Here are my notes on this process. I'm reasonably happy with the summary in terms of accuracy and conciseness, I'm now thinking of how to add value on top of this. Beyond just re-wording and re-summarizing, what would I want to see if I was reading a lit review? Probably a higher-level view, connections with other papers, etc.?
One thing to consider might be a section on gloss-free techniques, and tie it in with https://aclanthology.org/2023.acl-short.60/