scribe-org / Scribe-Data

Wikidata, Wiktionary and Wikipedia language data extraction
GNU General Public License v3.0
23 stars 25 forks source link

Create Swedish to all other languages translation process #79

Closed andrewtavis closed 6 months ago

andrewtavis commented 7 months ago

Terms

Description

The goal of this issue is to create a process whereby a single file is used to translate all words within Swedish/translations/words_to_translate.json to all other Scribe languages. To achieve this we'll be using m2m100_418M, with the output being a JSON file that has a string and keyed values for each language. This can then be transferred to an SQLite database table with each string in an index corresponding to a column value for each language.

Of specific importance is trying to get a metric of the accuracy of the translation and doing a cutoff such that we're no longer including low quality translations in Scribe applications :)

Contribution

Happy to work on this or support someone with interest in working on it!

Shorla commented 6 months ago

Hi @andrewtavis, My name is Olushola Ogunkelu. I am a new GSOC contributor. I went through this issue and I would like to work on it.

A bit of my background: I have experience contributing to open source and working with Python. But this is my first time working with machine translation.

I would appreciate some help on how to get started. Also, do I need to understand Swedish language to work on this?

andrewtavis commented 6 months ago

Hey @Shorla 👋 We'll be merging in an issue in a few days that will help you work on this. You'll be able to follow the code for the English translations that we have a PR from, and I'm sure that @henrikth93 would be willing to help us check the quality of some of the translations afterwards! I'll be in touch when you can start working on this, but for now I'll assign you 😊

Shorla commented 6 months ago

Thank you! I can't wait.

andrewtavis commented 6 months ago

Hey @Shorla 👋 The process has been set up and we're ready to implement here :) It's actually quite streamlined now. If you make a version of scribe_data/extract_transform/languages/English/translations/translate_words.py that replaces SRC_LANG with Swedish we should be good to go here 😊

Shorla commented 6 months ago

Thank you! I will get right to it

andrewtavis commented 6 months ago

Closed via #114 😊 Appreciate the support with this, @Shorla!