Additional font support needed for rendering some languages like Chinese, Hindi, Korean, Arabic, Farsi ...

Cervator commented 7 years ago

With #2666 merged we have Chinese translations but they do not render in-game (for instance in the language drop-down. I wonder if that's because there is a whole other Noto font family for that character set: https://www.google.com/get/noto/help/cjk/

If we need that to be able to properly display Chinese we may need to build in the possibility to do a full font swap when changing language. Although I wonder what that'll do for untranslated fields?



Related forum thread: http://forum.terasology.org/threads/dynamically-generating-characters-for-glyph-heavy-languages-i-e-korean-chinese-etc.1769/

Edit: Related tutorial https://github.com/Terasology/TutorialI18n/wiki

smsunarto commented 7 years ago

Added flag and change localisation file name to follow ISO 639-1 convention #2697

msteiger commented 7 years ago

I think that having this Chinese dropdown entry should be fixed/removed asap, because it's broken.

I'd like to add typescript support, if possible. What is the smallest subset of character glyphs? It looks like the Table of General Standard Chinese Characters still covers over 8000 glyphs. Do we need them all? Ping @smsunarto

smsunarto commented 7 years ago

@msteiger We won't need them all, chinese (Mandarin) doesn't use a system similar to alphabet hence their crazy amount of glpyhs since every word have their own unique glyph. Ideally we should only use the glpyh that is used by the translation. But I'm pretty sure this one should suffice https://en.wikipedia.org/wiki/List_of_Commonly_Used_Characters_in_Modern_Chinese really depends on whether or not the translator uses complex words though.

Cervator commented 7 years ago

Here's perhaps an odd thought: should we merge the outstanding languages even without proper font support and just leave them disabled in the drop-down somehow? So we can get the PRs out of the way, keep working on them via Weblate if needed, and focus down to a single effort on better supporting fonts?

msteiger commented 7 years ago

Sounds reasonable. I wonder whether it should be in the main project or should live in a module.

smsunarto commented 7 years ago

@msteiger I'm pretty sure glyph support is an integral part of the system. Im not sure how we can activate a new module outside of the world creation menu too.

MarkDerosier commented 7 years ago

To overcome this, we can first move some of the text into icons (a wrench for settings, a globe for language, etc), then use a simpliefied character set. (simplified Chinese vs traditional), and then to really save the space, all text is actually vectors.

Translate our words, load the font up into inkscape, make the labels, then convert font to path.

Now we're just loading a small SVG file instead of loading a massive font. A hack, but a good one.

We'd need SVG support in menu (which would probably end up getting us a better scaling UI anyways).

smsunarto commented 7 years ago

@MarkDerosier that would mean that we need to add new glyph every time we change/add new words? I like the text into icons idea tho. CC: @Cervator @msteiger

GabrielXia commented 7 years ago

Hello everyone !! I'm from China, I would like to help in this issue ! I've test in a simple module and it seems that our NUI doesn't Chinese character display. Could you please tell me where I could get start ? Thank you :-)

Cervator commented 7 years ago

Heya @GabrielXia and welcome to the glorious mess that internationalization appears destined to become! :D

I just merged your PR #2784 and all outstanding language PRs (closed a couple) and pushed to the weblate branch to thus update Weblate so you can see language at http://translate.terasology.org/projects/terasology/menu - only you can't see all of them, hmm. In-game the 5 incomplete languages are now excluded as intended so we have nothing broken going on.

Also managed to forget to first commit locally on Weblate and push out 5 commits with supposedly unique changes from there. In the interest of time I did a reset to get it fully up to date with develop, knowingly erasing some progress. Sucks but the amount of time it would've taken to fix would have been way worse than replicating whichever bits of translations we lost :/

So! Outstanding issues. I renamed the title of this to more accurately describe the challenge: showing advanced language fonts after picking one of the associated languages in-game. They include:

Chinese: Was already merged in the past. Needs font fix. Not showing in Weblate
Korean: #2745 - fixed flag, needs font fix. Not showing in Weblate.
Farsi: #2735 - right-to-left shows on the menu_fa.lang in IntelliJ, neat! But needs font fox. DOES show in Weblate
Arabic: Briefly came up in #2769 as it has no translations (was requested via Weblate). Needs font fix. DOES show in Weblate. Flag is tricky since Arabic covers many countries so there is no single flag and ar conflicts with Argentina. But we shouldn't need an icon for Argentina (Spanish is the official language) so I made a pure green flag_ar.png hoping that'll count as sufficient and non-controversial.
Hindi: #2733. Needs font fix. warn-modules claims to have an illegal escape character - huh? Hindi newline somehow? @VaibhavBajaj help :-)

There is a bunch of information in the linked PRs, especially from @msteiger having experimented with extra fonts with them showing in-game in at least two cases. But exactly how to bundle them is up for debate.

I would start by reading through all the links then maybe summarizing them further here so we have a central place to discuss all the related pieces together :-)

Additional:

We had a Turkish translation PR #2583 but it translated the translation keys verbatim, not their corresponding values in the English file. @tdgunes might be able to help redo that sometime :-)
The Indonesian entry in Weblate looks outdated - the actual menu_id.lang has translations where Weblate is showing identical untranslated string. Maybe we need to re-index something?
- @msteiger or @qwc could either of you please check to see if we can badger Weblate to properly index or see if we need to add config to have all the languages present and updated properly?
On text -> icons to translate less I realized a flaw in our cunning plan: another best practice is for icons to have tooltips to explain what they do ....
We need a wiki page on translation much like https://github.com/MovingBlocks/TerasologyLauncher/wiki/Add-New-Translation - we should add in the new exclusion thing from #2784 as anybody wanting to test an otherwise excluded language needs to comment out its exclusion entry in PlayerSettingsScreen.java - I left comments in there per language to indicate what is outstanding.

Had in mind to do a better summary of the options myself, but then suddenly it was nearly 3 am. Not sure how that happened!

VaibhavBajaj commented 7 years ago

@Cervator This particular problem has me confounded. I don't know exactly how I could help. Could you give me more details on the error? Thanks!

Cervator commented 7 years ago

This particular problem has me confounded. I don't know exactly how I could help. Could you give me more details on the error? Thanks!

Which error? :D I described several problems, hehe.

Weblate might just need some system administrator review to find out why it isn't showing everything at the latest level. When it is translation for anything remaining should be easy.

Adding support for more fonts is beyond me, hoping @msteiger can help out there as well as others able to research some of his initial findings. I don't know how to guide there better, @VaibhavBajaj :(

syntaxi commented 7 years ago

Reviving this in light of the Arabic pull.

As a solution, can we say ship with the current Noto font we have and then when a player enables a language like Chinese or similar that requires a separate font file we download it then. If there is no translation we fall back onto the default font?

OmarAglan commented 7 years ago

hi, my name is Omar you can add font like (Arial) and its support Arabic. I am going to complete my translation later on thanks

Cervator commented 6 years ago

Belated hello @Omaranwa - there is a Noto font set that does support Arabic and all kinds of other fonts. The problem this issue is meant to cover is that our engine at present doesn't have the logic to switch the active font at runtime in response to a player changing to a language requiring extra characters :-)

Cervator commented 5 years ago

https://forum.terasology.org/posts/16087 - Conlang / custom script feedback and polish

MovingBlocks / Terasology

Additional font support needed for rendering some languages like Chinese, Hindi, Korean, Arabic, Farsi ... #2677