encdoing issue - Githubissues

106AbdulBasit commented 1 year ago

I have downloaded the test .text file from the below link, which is basically the test set of eng- urdu,

The file has the following like pairs

Franco has blue jeans. ÙØ±Ø§Ù†Ú©Ùˆ Ú©Û’ Ù¾Ø§Ø³ Ù†ÛŒÙ„ÛŒ Ø¬ÛŒÙ†Ø² ÛÛ’Û” Ù„ÛŒÙˆÙ† Ú©Û’ Ù¾Ø§Ø³ Ù†ÛŒÙ„Ø§ Ø¬ÛŒÙ†Ø² ÛÛ’.

which encoding will be used, I have used "utf-8", "cp1256", "iso-8859-6" and many more, not able to process these lines,

jorgtied commented 1 year ago

It should be in UTF-8.

106AbdulBasit commented 1 year ago

Surprisingly when the file was opened in the Mac it was doing well .

Helsinki-NLP / Tatoeba-Challenge