search

mozilla / translations

The code, training pipeline, and models that power Firefox Translations
https://mozilla.github.io/translations/
Mozilla Public License 2.0
154 stars 33 forks source link

Made-up words in translations #843

Open eu9ene opened 1 month ago

eu9ene commented 1 month ago

We recently got a report about made up words in Turkish.

I also tested the new ru-en model and noticed a lot of non-existent words there as well.

For example: https://habr.com/ru/articles/842924/

The title is translated as "Language environment and teacher-sdocument. Non-evous moments". It should be "Language environment and native-speaking teachers. Non-obvious points".

We should investigate why this happens. It might have something to do with the recent robustness fixes or using a shortlist.

eu9ene commented 1 month ago

Translation by the teacher model: "Language environment and native teachers. Non-obvious moments".

I don't see any made-up words in the full text as well even though the fluency is far from perfect. Only a couple of informal abbreviation were translated as is, for example "дз" (home work) as "DZ" and "выпускники инязов" (graduates of universities that specialize in foreign languages) as "graduates of inaz". This text is written in a very informal style, so I guess overall lower quality of translation is expected.

This means we have the issue with made-up words specifically for the student model.

eu9ene commented 1 month ago

I don't see any made-up words in translation of news where more formal language is used.

gregtatum commented 1 month ago

I wonder if this could be caused by the decoder being too shallow, or the decoder not being big enough. This could be good to experiment with, and also test the difference in the performance of the models.

marco-c commented 1 month ago

Maybe lexical shortlisting could also be affecting this?