search

Niger-Volta-LTI / yoruba-text

Yorùbá language training text for NLP, ASR and TTS tasks
GNU General Public License v3.0
72 stars 25 forks source link

Separate òwe corpus into EN & YO #16

Closed ruohoruotsi closed 5 years ago

ruohoruotsi commented 5 years ago

Separate òwe corpus into individual EN & YO files.

The format typically follows this repeating pattern:

YO
EN1 - literal translation
EN2 - proverbial (non literal) translation
EMPTY LINE

For example:

A di gàárì sílẹ̀ ewúrẹ́ ńyọjú; ẹrù ìran rẹ̀ ni?
We prepare the saddle, and the goat presents itself; is it a burden for the lineage of goats?
Goats that know their place do not offer their backs to be saddled. This is a variant of A gbé gàárì ọmọ ewúrẹ́ ńrojú . . .

A fi ọ́ jọba ò ńṣàwúre o fẹ́ jẹ Ọlọ́run ni?
You have been crowned a king, and yet you make good-luck charms; would you be crowned God?
Being crowned a king is about the best fortune a mortal could hope for.

A fijó gba Awà; a fìjà gba Awà; bí a ò bá jó, bí a ò bá jà, bí a bá ti gba Awà, kò tán bí?
By dancing we take possession of Awà; through fighting we take possession of Awà; if we neither dance nor fight, but take possession of Awà anyway, is the result not the same?
Why make a huge production of a matter that is easily taken care of?

How the text is separated is up to you. One idea is to have 2 files for each òwe: yo_001.txt, en_001.txt (which includes both translations on 2 different lines as above ☝️

Then for text preparation, we can just aggregate all the YO text. For Machine Translation, we can create a parallel YO-EN corpus by taking the literal or proverbial EN translations.

Please feel free to add any questions or clarifications.

Olamyy commented 5 years ago

Thinking about this, would a CSV not be a better representation of the data than a text file? We can have a csv file with the format: |yoruba|english| translation

I suppose the challenge with that would be updating the file when we get more data. Would be easier to dump into a folder than having to update a csv each time.

I'll go ahead and implement your idea now. If the csv idea appeals enough, we can always have that implementation available later.

ruohoruotsi commented 5 years ago

I think the problem with a CSV |yoruba|english| translation is not that its not a good idea or intuitive format ... it is that that is not the format used by OpenNMT, Tensor2Tensor etc.

See here (which is also how we do it for ADR), how they split text into "source" (langA) and target (langB) files ...each line corresponding to the same line in the other file: http://opennmt.net/OpenNMT-py/options/preprocess.html

So if you use a CSV, you still have to segment it into sources & targets before you can create an input-training batch, that's why I was thinking if we do any work to split it, why not make it ready for either a monolingual YO ADR task as well as a NMT task ...any sha, that's my thinking.

Back to CSV, another idea, is to ALSO make a CSV that can be used for human validation of the text or another non-NMT, non-ADR purpose.

For maintenance/update, I haven't found a good way update text, since we get text submissions periodically. For Kọ́lá's and Toluwase's texts, what I did was concatenate all texts (new & old) then did (on Linux) uniq | sort to filter out the duplicates and sort them (nice but unnecessary for training).

Let's feel free to establish our own "best practise" for our specific computing tasks!