glosbe cleaning

[chaoSefat]

Texts scraped from glosbe have English artifacts that reoccur in text. Which includes:

• "Translation of $Page_name into $lang_name"
• "$word is the translation of $page_name into $lang_name"
• "Sample translated sentence:"
• "\n$Page_name\nnoun\ngrammar\n+\nAdd translation\nAdd\" ...... ...... And Much more.

Being a dictionary it provides translation from a language to the other. For example "https://glosbe.com/en/ada/yard" would translate yard from English to Adangme. Another page "https://glosbe.com/fi/ady/koskea" would translate Finnish word "koskea" to the language "Adyghe". Furthermore, they provide example sentence translated.

Possible Options:

• Don't include glosbe text for language modeling.
• Clean up. Might be extremely convoluted due to the inclusion of other languages in a way which is hard to separate.

[chaoSefat]

Perhaps a general text cleaner can be and should be written that would clean high resource languages out of scraped text rather that a site specific cleaner.

[chaoSefat]

Beta cleaner made on colab. Can be integrated when text dataset is formed.

[chaoSefat]

https://github.com/fedelopez77/langdetect