Localization into other languages

[davidjoffe]

Localization into other languages

[This stuff relates to the new 'unstable/dev' stuff that's not yet enabled in the core stable codebase] These two issues relate to one another: https://github.com/davidjoffe/dave_gnukem/issues/178 https://github.com/davidjoffe/dave_gnukem/issues/148

Some quick off the cuff thoughts, and regarding the beta sdl2-ttf stuff recently added (but not yet enabled): This is something I'd rather try do "carefully" with a game that has a specifically retro vibe, with I think several separate goals to keep in mind with looking forward also:

(Also I don't want to rapidly disturb and break existing ports by adding dependencies - at some point will want to phase this stuff in .. so not sure yet if should be somehow 'opt-in by downstream' e.g. maybe by differernt Makefile or Makefile flags or if should be for something called 'verison 2' or 'version 1.2' or what, don't know yet)

I do want to be able to make 'modern, high-res nice-smooth-visuals' and 'all languages' etc. games potentially off codebase, but:

"Dave Gnukem 1"

While it may look ugly from a 'modern perspective' the game's 8x8 retro font is I feel part of the "90's retro vibe"

Also the game itself has now been released as version 1 with a certain set of levels, other than maybe adding separate level packs or "bonus" level packs to future versions I think the core game should probably not really change again per se (? unless it's some parallel universe type of release with just a different set of levels with same game core) but the POINT I'm trying to make though is that one thing we CAN change, that adds a lot of value potentially to many more people, is that we can add e.g. localizability (i.e. abililty to have the game run "in" different languages, and be able to e.g. enter Unicode names in high score lists which I've recently implemented in an unstable-dev feature branch I merged in but the stuff is not yet enabled - enable the Unicode option in config.h to try it out).

However this is a complex topic as e.g. if we switch to modern font rendering needed for many languages it may lose some of the retro vibe. So I think each localized port could/should perhaps have the ability to carefully select different desired fonts that could be also potentially "retro" For example the attached is (in fact, though it doesn't look like it) a special retro TTF font that is designed to look like 8x8, but would more easily allow us to, say, make a French localized version but still have some retro vibe (but without the menu gradients)

*[NB Yes I know translations in screenshots are wrong! Just some quick demo screenshots!] of a quick n dirty test here using a retro 8x8 ttf font (this would make it nice and easy to say make localized versions for e.g. Latin-alphabet-based languages at least - hmm like French or German - and still have a retro look and feel but is less work than trying to add all the diacritic/accent characters to the current old 8x8 gradient font):

"Dave Gnukem 2" or "Other potential games off codebase (retro or modern)?"

If we create any derived games off the codebase it no longer has to follow the "retro vibe" thing that i think "Dave Gnukem 1" must preserve, but a hypothetical entirely different either 'Dave Gnukem 2' or 'any other game' wouldn't need to.

Help would be needed with actual translations by human translators (machine translation is not good enough for final 'quality production' release, but perhaps if it's labelled beta in a free game some users could opt into it somewhere with the understanding it's not final translations)

[davidjoffe]

One thought, I think when we localize we should try divide strings into something like two main subsections, which could live in slightly different place, maybe even different repos:

(1) Generic strings that could be used in any game (e.g. "Quit" or "Begin game" or "Settings" or "Redefine keys") (2) Dave Gnukem specific strings (e.g. the plot, or instructions)

That would make it easier to re-use the code for other games too

[davidjoffe]

Copying and pasting some 'thinking out loud' comments of mine here from another thread below:

(Technically: The current main 8x8 game font doesn't support e.g. French diacritics, as it's an ancient non-Unicode raster font I did largely by hand centuries ago around the time Stone Age people were first creating ASCII, I had a look in Photoshop and concluded it would be "probably too painful" and not worth it to try adapt the current 8x8 font to add diacritics - BUT long story short, I found the quite above nice looking open retro font that does seem to have the necessary diacritics for French - it's TTF and I was first thinking I could use my new TTF code (but that would add a new dependency, creating some maintenance work for porters & eg Debian package management so ideally I don't want to add new dependencies if I can avoid it), but then I thought I could also alternatively just pre-render into a raster font all the characters needed for French and other major Latin-alphabet-based languages and then posssibly with zero new dependencies but just some work on the coding backend for utf8 in the raster font rendering stuff and then to add support for loading translations and some UI code for selecting language we could do this, that is some fun 'weekend coding'.

I'm keen to do coding changes to support localization (if anyone wants to help they can), but would need human translators to do actual translations!

[davidjoffe]

I know translation is a lot of work, especially if you want to do it well, but while I am happy to do any coding changes unfortunately (though I speak some bits of many languages) I'm not qualified to translate into any of them.

As this is open source there's no budget to pay translators (unless we get major sponsorship/donations) so translation work would currently have to be sort of 'open source voluntary' type of thing.

Also NB Google Translate (and other machine translation) is of course not good enough to be suitable substitute for actual game translations - I grabbed some just to test the coding changes as a sort of 'proof of concept' but to do quality localization we'd need human translators.

[davidjoffe]

If anyone is interested in helping localize/translate this game into any language please let me know here?

[davidjoffe]

For languages like Arabic we'd likely need to use TTF ultimately for the complex text shaping ... but to start with we could potentially have a mix of approaches in the code where if a language doesn't require TTF I want to allow something like the abovementioned raster-based solution.

In future we could/should also support high-res fonts for high resolutions etc. for derivate games but for now this is a retro game and I want to try keep it having a 'retro look and feel' even if localized (that may be debatable and perhaps that could maybe ultimately be a setting in the settings for the game, and localizers may also need a bit of freedom to adapt & choose best solution for each language but anyway.)

[davidjoffe]

Small note specifically regarding Arabic support: The game's new libsdl-ttf code I've added in the 'development branch' (i.e. djUNICODE_SUPPORT and/or djUNICODE_TTF config.h preprocessor config.h options) supports Arabic in the 'feature branch(es)' for Unicode etc., I've added this, but it requires the "harfbuzz extensions for libsdl2", which are only available on some platforms/configurations etc, depending on exact version of libsdl also.

[jmfergeau]

you can count me in for the french translations ^^ I just need the files for the translations. :p

Will you use something already existing for it like po/mo files or another method?

[davidjoffe]

Awesome! That's exciting to hear .. I need to do a little work on the coding side but keen to do that hopefully by/during this weekend.

Not 100% sure yet which format, maybe .po, busy thinking about it, may play around a little then decide very soon.

[davidjoffe]

Unfortunately not going to be quite ready this weekend :/ - but should be soon!

[davidjoffe]

@jmfergeau Sorry for delay, have been busy but have made progress on this and almost ready to have an actual French Dave Gnukem, are you still possibly available and interested to help with translation?

I've added a new command line option "-lang" that allows one to select a different language on startsup, e.g.:

./davegnukem -lang fr

I've also added a whole bunch of other required changes, including PNG support (and jpg etc.), raster font helpers, added Pixel Operator font, some utf8 stuff and other Unicode-related.

When you select "-lang fr" then it loads a different menu font, which should support French accents e.g.:

https://github.com/davidjoffe/gnukem_data/tree/main/fonts/pixeloperator

It then also displays the selected language in the bottom right here:

Now, next step is just to start adding translations! I need to do a bit more work to unhardcode strings and that's the next step.

[davidjoffe]

Doing a bit more work on it here and got this proof-of-concept basic main menu translation working :) :

NB warning the below screenshots are "Proof of Concept AI-done Translations" to help with development and testing, not human translations:

davegnukem -lang af davegnukem -lang es davegnukem -lang de davegnukem -lang fr etc.

Ready to start adding and selecting French too:

BUT the next step is more game strings need to be extracted into a localization format first (not sure yet but currently leaning toward .po files)

Also there are still a few game strings "baked into" the main game 'skin' e.g. "Health", "Score", "Inventory" I need to just un-bake to make localizable.

I see localization as do-able in a few 'phases', this is phase 1 in progress

The above is not yet in the RELEASE Dave Gnukem, but it is basically currently in the latest GitHub main.

[davidjoffe]

Good news, the latest 'main' now has the first basic localization enabled, and the first basically-working po files in a new 'locale' subfolder (though very few strings are still localizable, we can start accepting pull requests or direct contributions for translations on these files, e.g. fr.po should now be working for holding the official French Dave Gnukem translations :)

Translator workflow could be to (1) fork the main repo, (2) edit .po files and (3) submit PR's and I can merge them in, would that work? Should we try also integrate localization platforms that automatically integrate with git?

To test localization one can do something like:: (1) 'git clone' the repo (2) cd into the repo folder (3) ./getdatafiles.sh (4) make -j8 (5) ./davegnukem -lang fr

That should (provided one has the latest 'data' also) run DG, loading the 'locale/fr.po' file (though currently the string translations aren't there yet)

I will need to add more strings though, currently it's only basically the main menu and a few on-screen strings e.g. "Score" "Health" "Inventory" etc.

Would be great if the 1.0.4 release could have French

I'm wondering how we should we try license actual translations? I'd really like to try lean towards either by default 'MIT' or public domain to facilitate re-use in both possible commercial and open source? If a specific translator really disagrees for a particular language one could perhaps use something else. (I am thinking about, I maybe want to in future make non-free games off of the source code as it's a lot of work to make this game, and I can't work for free on all this, and the donations are very very few, so I'd like to be able to use the translations if possible in that to help support the development, but this DG will always be open source.)

It would be awesome if the 1.0.4 release could have at least French (and perhaps Afrikaans) localization.

I'm using 'pgettext()' (or maybe also 'gettext()') in the source code itself for localizable strings, and use 'cd src && xgettext -c -C *.cpp' or some-such to generate the main .po) so I/we need to add more pgettext() calls. (In fact, that standard is from 'GNU gettext()' which ironically is the same GNU in the 'Start Gnu game' pun in the main menu)

[jmfergeau]

sounds good. I know well PO/MO files and translation, it's a very effective way for localization ^^

I'm gonna start on the translations right away. i keep you in touch very soon ;)

[jmfergeau]

After tests, it seems to work like a charm :)

Since only this menu has translation lines, I guess i'll have to wait for an update on the PO file to translate the other lines. Anyway, I'm gonna make a PR with it so far. ;)

[davidjoffe]

Love it! Just merged it. Very excited to possibly have a French Dave Gnukem - will add some more strings soon :)

Some things are easier to re-code/restructure than others, hm, will start with low-hanging fruit

I've also added more changes for detecting/supporting RTL (Right-to-Left) languages for game UI