Multilanguage game support

[Ghabry]

The Player should be able to load game translations on-the-fly without embedding them in the database or maps.

See also LcfTrans in the Tools repo.

Probably will be also useful for @pureexe and @whateverzone when Thai is fully supported.

Work plan (should be discussed):

• Place translations in "translation/[Translation-Name]", file hierarchy is mirrored (RPG_RT.po, Map0001.po).
• The Translation folder can also contain asset folders (ChipSet, Pictures, ...). They are prefered over the original ones when available
• The Translation files will be UTF-8, no encoding issues \o/
• Language selection on game startup, language name extracted from "Language:" in RPG_RT.po

[pureexe]

WOW!! it's great. but i have a question. @Ghabry How can i create translation file? (.po extension)

[whateverzone]

That's great news! Thank you a lot! All the plans sound convenient though I have the same question as @pureexe and I also need to hear more details about these.

[Ghabry]

Don't be too excited yet. It is still in the planning phase.

You can create ".po" files with the tool "LcfTrans" which will be in our tool repository soon. Simply invoke it in the game directory and .po files are created for Database and Maps.

Po is a popular file format used to create translations, especially open source projects use it. The content is basicly

msgid "Original text"
msgstr "Translated text"

About the loading in EasyRPG Player: No work was invested into this yet. I plan this for the 0.5 release (coming this year). But this will solve all encoding problems because the files are in unicode.

[pureexe]

Thank you for describe. I hope this feature will release soon.

[Ghabry]

Another open problem is how to handle text that is too long (more then 4 lines of text). Probably just insert more lines and the Player will handle it gracefully...

The other direction is a different issue (from 2 message commands to 1 because the translated message is shorter)... Probably some in-band-signalling "\0"-char... Will document it when I have an idea.

[sorlok]

Good evening,

This Issues looked really cool, so I implemented a hacky test branch to help with the discussion: https://github.com/EasyRPG/Player/compare/master...sorlok:snh_translate?expand=1

Most .po text works, including Terms and Message Boxes. Image swapping also works, using the directory structure discussed earlier. Screenshots:

Some notes on my implementation:

• I used tinygettext for simplicity; I don't use any complex features, so we could swap it out.
• I use a "Translation" object stored in player.cpp, and a wrapper "Tr" namespace. This means you can use synax like the following:
  • Tr::Term("New Game"); // Translates a "Term" from RPG_RT.po
  • Tr::SkillName("Fire"); // Translates a "skill.name" from RPG_RT.po
  • Messages must remain <= 4 lines for now; it should be fairly easy to extend them later. They are currently truncated and print a Debug log line.
• The Images are switched by having FileFinder::FindImage() search for the translated image first, and the non-translated image second. This is fast (it uses FindFile with some modifications), and is eventually cached by the Cache.
• I force the language to "es" (Spanish) in code; you can't switch the language right now.
• I tested various language fonts (see Title screenshot) --things seem to work fine, but some Spanish letters only work if I force the file to "UTF-8" encoding.
• I don't have a solution for translating EasyRPG-specific strings (like the File Browser), but I think that's outside the scope of this Issue (which is mostly for games).

I have a lot of design questions:

• Which gettext library do we want?
• The "Tr::Term()" syntax needs to be decided upon early, since it's widespread in the code base. Please let me know what you prefer.
• Does it make sense to switch Images in FindImage? Also, how does the emscripten stuff (async_handler) work? I think my code won't work with Emscripten as-is.
• Can you elaborate on how you think languages should be selected or switched? Will there be a new item in Scene_Title called "Language", or will we try to auto-determine the language from the user's Locale (or something else)?
  • In particular, how will the user switch languages? Is there some kind of in-game "Options" menu?
  • Do we store the current language in the Save file? This would allow me to play in English and you to play in Spanish on the same game.
  • How will we get the language name (e.g., es => "Spanish")? You mention storing it in RPG_RT.po, but it seems wasteful to have to read the .po file just to populate the Language menu (what if there's 20 languages for a given game?). Maybe instead have a "Language.Spanish" file in the directory, with a fallback to some well-known lookups?
  • Note that the Translation class should be able to switch a language at runtime, but we'll have to invalidate the current Cache'd images.
• Right now, I have to create a tree scanner to find all the "MapXXX.po" files. I think it might be better to rely on the FileFinder's "sub_members" and just look up map files as-needed at runtime.
• "skill.using_message1" is incorrect for RM2K games unless they are RPG2KE. Basically, LcfTrans generates a msgid like "%S uses poison", but the %S is stripped at runtime. LcfTrans should instead generate " uses poison" for matching to work right.
• I think LcfTrans should force a unicode character (>0xFF) into the generated text file. This will force most editors to save as UTF-8. (Some "smart" editors try to convert to some ANSI nonsense if it detects no "obviouysly" unicode characters).
• The "Message" code can pull from RPG_RT_common.po, RPG_RT_battle.po, or MapXXXX.po, depending on who is showing the message box. I don't know how to detect if we are in battle or in a Common Event, so I just do redundant scans (and cast the game_interpreter to game_interpreter_battle). What's the best way to detect if the current interpreter is in battle or in a common event vs. a map event? If there's no good solution, we'll have to rework the .po file structure.
• Do we have to consider the RTP at any point? I.e., if translating a game made with the English RTP into Japanese, do we need to hot-swap out to the Japanese RTP? Or is this not a problem with EasyRPG?
• I think we should deal with the following things later: (a) trimming message boxes that are too long and (b) adding new message boxes if the resulting translation is >4 lines. (The initial code will be complicated enough.)
• I think it would be easiest to develop this feature if someone was working on an active translation to help stress the code. I used The Blue Contestant, but maybe the demo game would be better?

Well, this was a long comment, but I hope this got things going in a good direction! I'm very impressed with the LcfTrans tool, since it seems to mostly do everything we need as-is. The .po contexts are well-structured.

Also, just to be clear: this branch is very hacky and should not be merged. But it does work, so now I understand the scope of the needed changes better.

PS: All the language stuff is Google Translate; I don't speak any other languages.

[Ghabry]

@sorlok This is awesome! I havn't tested this yet but this must have been lots of boring work to get all the string subsitutions hooked in.

Please open a Draft PR for it to allow further discussion there.

It is not really necessary to use Tinygettext, the translation.cpp/translation.h already has a simple "fromPO" function. You could the translation class for parsing (just copy the code you need over into Player/translation.cpp)

https://github.com/EasyRPG/Tools/pull/21

[Ghabry]

Well, lets respond to your wall of text :)

Which gettext library do we want?

Because parsing Po is not a difficult task imo the translation.cpp from LcfTrans is good enough. This also makes it easier extensible. Currently reading games from ZIP files etc. is planned so there must be support for a file stream interface.

The "Tr::Term()" syntax needs to be decided upon early, since it's widespread in the code base. Please let me know what you prefer.

This Tr::CONTEXT(STRING) looks like a sane API to me.

That's a lcftrans problem: I'm not sure yet if all the terms should be just in a context called "term". Maybe every term should have its own context: term.yes, term.no, term.new_game. Is unlikely to have the same term twice (and when they conflict you are out of luck right now)

(Assume there is a game where two terms have the same string but in the target language not, this must be solvable)

For e.g. actor names I like the current approach but the lookup should be made smarter: The context can stay "actor.name" but there could be also a lookup for "actor.name.ID" before, so for actor 2 the lookup order is "actor.name.2", then "actor.name". Same for other objects.

Also interesting are event messages when a split is required (currently same messages are merged to one PO entry) but message processing is not very often used in our code (only once?) so thinking about this later is fine.

Does it make sense to switch Images in FindImage? Also, how does the emscripten stuff (async_handler) work? I think my code won't work with Emscripten as-is.

Yes the "redirection" should happen in the Find*-functions. For Music this is not very useful, so just images is okay for now.

Ignore emscripten for now, is not easy to solve there so would postbone this until the rest is clear.

Can you elaborate on how you think languages should be selected or switched? Will there be a new item in Scene_Title called "Language", or will we try to auto-determine the language from the user's Locale (or something else)?

When there are translations available add a "Language" entry to the Title screen.

Other ways:

• Command line option "--language=es".
• Through our config scene (in the works)

Do we store the current language in the Save file? This would allow me to play in English and you to play in Spanish on the same game.

Give this a low priority for now. The prefered language could be a global setting (as said above config scene still in the works :)). Storing the current language in the save file sounds interesting, at least would be easy to add through liblcf and a new field.

How will we get the language name (e.g., es => "Spanish")? You mention storing it in RPG_RT.po, but it seems wasteful to have to read the .po file just to populate the Language menu (what if there's 20 languages for a given game?). Maybe instead have a "Language.Spanish" file in the directory, with a fallback to some well-known lookups?

The translations should be stored in "languages/ARBITRARY_NAME". The FileFinder recurses them and maybe looks for a "Translation.ini" file that contains the metadata of the translation? Metadata TBD, one is the name of the language.

Note that the Translation class should be able to switch a language at runtime, but we'll have to invalidate the current Cache'd images.

Yeah just invalidate them. Some of them will be still outdated until a map changes but this is good enough - Many programs require a restart when you switch a language.

"skill.using_message1" is incorrect for RM2K games unless they are RPG2KE. Basically, LcfTrans generates a msgid like "%S uses poison", but the %S is stripped at runtime. LcfTrans should instead generate " uses poison" for matching to work right.

My idea here was that when translating you usually have to adjust the word order, so the %S etc must be supported in all versions but I see your point here......

The "Message" code can pull from RPG_RT_common.po, RPG_RT_battle.po, or MapXXXX.po, depending on who is showing the message box. I don't know how to detect if we are in battle or in a Common Event, so I just do redundant scans (and cast the game_interpreter to game_interpreter_battle). What's the best way to detect if the current interpreter is in battle or in a common event vs. a map event? If there's no good solution, we'll have to rework the .po file structure.

Not sure. Need to check this.

Do we have to consider the RTP at any point? I.e., if translating a game made with the English RTP into Japanese, do we need to hot-swap out to the Japanese RTP? Or is this not a problem with EasyRPG?

The Player already does lots of magic to redirect RTP access and you can't translate filenames with lcftrans, so I don't an issue here.

I think we should deal with the following things later: (a) trimming message boxes that are too long and (b) adding new message boxes if the resulting translation is >4 lines. (The initial code will be complicated enough.)

Yeah this is hard and can be skipped for now.

I think it would be easiest to develop this feature if someone was working on an active translation to help stress the code. I used The Blue Contestant, but maybe the demo game would be better?

There are various games with translations but this would need some scripting to merge two LDBs in a single PO... But for marketing reusing an existing translation sounds like the best idea... hmm

E.g. Yume2kki is always many versions behind with the English translation because it is so hard to do it. With LcfTrans it would be very issue (the tool just needs further logic to extend existing PO files with new strings...)

[sorlok]

I made PR #2287 as requested (I'm still reading through your responses; thanks for the detailed feedback!)

[Ghabry]

Fixed by #2287

[Ghabry]

Another happy customer:

Jan (Deep8) wanted to move from translation through branching on a variable to lcftrans.

Problem: This game uses the DynTextPlugin (and a custom player).

So I added now support for translating @write_text and @append_line 😅 (well and dumping of it to deep8trans)

I'm kinda surprised how well that works. Good translation API :)

https://github.com/DeepIA8/easyrpg-player-deep8/commit/d6ce3fa0dc2a7a7fffd114a5bb06657ce0bcef23

[sorlok]

Another happy customer:

Jan (Deep8) wanted to move from translation through branching on a variable to lcftrans.

Problem: This game uses the DynTextPlugin (and a custom player).

So I added now support for translating @write_text and @append_line 😅 (well and dumping of it to deep8trans)

I'm kinda surprised how well that works. Good translation API :)

DeepIA8@d6ce3fa

Glad to hear it! Always nice when an API decision turns out to have been the right one.

[sorlok]

Hmmm, quick follow-up on this change:

    if (!current_language.empty()) {
        // We reload the entire database as a precaution.
        Player::LoadDatabase();
    }

The reason I didn't check for "empty()" is in case the player changes the language to, e.g., "French" then back to "Default". In that case, I think the current_language string will be empty but the DB will still need a refresh.

[Ghabry]

This is just a temporary workaround: The language changing is in Deep 8 on a map and reloading the database makes the tilemap pointer of the map a dangling pointer and crashes the engine.

So this "fix" allows to change the language once before crashing. Have to fix this properly, e.g. some "OnLanguageChange/OnDatabaseReload" callback to all scenes or something like this.

Or always refetching the pointer from the database. Not sure yet

[sorlok]

Ah ok, got it!