search

ryzom / ryzomcore

Ryzom Core is the open-source project related to the Ryzom game. This community repository is synchronized with the Ryzom Forge repository, based on the Core branch.
https://wiki.ryzom.dev
GNU Affero General Public License v3.0
333 stars 90 forks source link

Support for utf8 with _words translation files #128

Closed ryzom-pipeline closed 7 years ago

ryzom-pipeline commented 10 years ago

Original report by Jan Boon (Bitbucket: [Jan Boon](https://bitbucket.org/Jan Boon), ).

Currently these files are in utf-16, but mercurial only supports utf-8 for merging. We should support utf-8 as file storage for these files. Other translation files are in utf-8.

ryzom-pipeline commented 8 years ago

Original comment by Meelis Mägi (Bitbucket: [Meelis Mägi](https://bitbucket.org/Meelis Mägi), ).

CI18N::readTextFile() which uses CI18N:: readTextBuffer() already supports utf-8 input (with bom).

It needs to be decided if text without BOM should be read as binary or utf-8. If it's going to be utf-8, then forceUtf8 probably becomes redundant.

There is also CI18N::writeTextFile() which needs to be decided if UTF-8 needs bom or not (or if we need to write utf-16 in the first place)

ryzom-pipeline commented 8 years ago

Original comment by Cédric Ochs (Bitbucket: [Cédric OCHS](https://bitbucket.org/Cédric OCHS), ).

Good idea :) I suppose this decision has been taken because Windows natively supports UCS-2 (almost like UTF-16) so perhaps it's faster to load them :)

ryzom-pipeline commented 8 years ago

Original comment by Cédric Ochs (Bitbucket: [Cédric OCHS](https://bitbucket.org/Cédric OCHS), ).

I'll try to load UTF-8 words txt files, but I suspect it'll work without any change in code or perhaps only in translation_tool :)

ryzom-pipeline commented 8 years ago

Original comment by Meelis Mägi (Bitbucket: [Meelis Mägi](https://bitbucket.org/Meelis Mägi), ).

make sure you use french or russian translations for test ;-)

ryzom-pipeline commented 8 years ago

Original comment by Cédric Ochs (Bitbucket: [Cédric OCHS](https://bitbucket.org/Cédric OCHS), ).

Sure :p But it'll be easier to compare changes in Mercurial :)

ryzom-pipeline commented 8 years ago

Original comment by Cédric Ochs (Bitbucket: [Cédric OCHS](https://bitbucket.org/Cédric OCHS), ).

As said Nimetu, it works fine apparently :)

We'll just need to (slightly) update translation_tools.

And yes, I think we could consider 8 bits text files to be UTF-8 by default if not having any BOM.

ryzom-pipeline commented 7 years ago

Original comment by Cédric Ochs (Bitbucket: [Cédric OCHS](https://bitbucket.org/Cédric OCHS), ).

Fixed: Translations files are all using UTF-8 by default now (and LF instead of CRLF), fixes #128