Update data importer to support CJK

mozilla / translations

The code, training pipeline, and models that power Firefox Translations

https://mozilla.github.io/translations/

Mozilla Public License 2.0

155 stars 34 forks source link

Closed eu9ene closed 2 weeks ago

eu9ene commented 1 month ago

The goal is to treat the language code zh as Mandarin Chinese in Simplified script for now.

convert all Chinese to Simplified (support for Traditional will be handled separately in #896)
add cjk augmentation modifier

closes #752 closes #741

eu9ene commented 3 weeks ago

@gregtatum I addressed all your comments so it's ready for another pass