Make translatable

[Epicpkmn11]

Edit: This is now more or less complete, redoing PR comment

• Makes more or less everything translatable
  • I used a #define based system similar to TWiLight Menu++, you define a string like STRING(TEST, "test") then use it in the code STR_TEST
  • This allows for the compiler to ensure you're referencing a real string and string names don't need to be included in the binary, only pointers
• Adds a language selection menu that is shown on first load and accessible from the HOME menu
  • This shows all TRF files in 0:/gm9/languages using their proper names
• Uses a custom TRF (Translation RIFF) format similar to the FRF font format for efficient storage and loading of translations
  • A script is in utils for converting from JSON, note that TRF files must be in the exact same order as the strings are defined so any update that changes the order should bump the TRANSLATION_VER define
  • They do not need to contain every string however, so old files will still work if the version hasn't changed and the English TRF can be an empty placeholder
• A Crowdin project has been set up for easy management of translations in JSON format
• Adds a README section on fonts and translations
• Adds myself to the README credits
• Closes #766
• Companion PR: #802

Original message

Currently only the start of `godmode.c` is done as an example and with a hardcoded Japanese translation. If this seems good I can continue on it, might take a bit since there are a lot of strings but it's mostly just copy string to `.inl`, give it a name, use STR_name in the source. Only a couple places have actually needed any real code changes where I changed it to avoid concatenation (always better to use a format string, never assume all languages have the same sentence structure as English) or maybe a couple other similar changes. For using the strings I did it like TWiLight Menu++/GodMode9i do where you define as string as say `PATH_FIRM_TOO_BIG` in the `.inl` file, then to use it you simply do `STR_PATH_FIRM_TOO_BIG`. If preferred I could also do it like `getString("PATH_FIRM_TO_BIG")` or such, I kinda like the way TWiLight does it because it means you get a compile error if you misspell a string instead of it simply not showing up or having a runtime warning. I added `-Wno-format-nonliteral` and `-Wno-format-security` to the Makefile since they were throwing a lot of warnings and I wasn't sure a way to actually fix them, there might be a better way to fix that though. > ```c > source/godmode.c: In function 'BootFirmHandler': > source/godmode.c:62:9: warning: format not a string literal, argument types not checked [-Wformat-nonliteral] > 62 | if (verbose) ShowPrompt(false, STR_PATH_FIRM_TOO_BIG, pathstr); // unlikely > | > ``` Edit: Fixed `-Wformat-security`, I don't think `-Wformat-nonliteral` is really fixable though since well, the whole point is that we're formatting non-literals as defined by the translation. - Closes #766 - Companion PR #802 (Note: The Japanese translation isn't usable with the current default font. Use [misaki gothic](https://github.com/d0k3/GodMode9/blob/master/resources/fonts/font_misaki_gothic_8x8.frf) or [GodMode9i](https://github.com/DS-Homebrew/GodMode9i/blob/master/resources/fonts/default-6x10.frf)'s font to test it properly, I'll probably copy the additions I made to GodMode9i back over to here later)

[d0k3]

Alright, I had a look. Sorry this took a bit. I'll still have to discuss this with @Wolfvak and @aspargas2 (you may want to chime in here). Looks good, though.

For now: It's absolutely important you make 100% sure no english strings are changed by accident in the process and nothing breaks. For english, this will be a pure under-the-hood-change. Exceptions: If you find typos in the process (of course). Do you have anything to double check the required manual changes or even, automate them?

Also, speaking names are important (which. Meaning: From the strings name, I should be able to somewhat conclude what's behind them. Maybe some scheme, which would convert the strings content to a define would make things easier. With all the strings scattered among all these source files, that may be a bit of work.

[Wolfvak]

This looks like a good start, but I have to mention a few things:

• The GM9 binary is already pretty large and adding more langs on top is only going to make it larger, so decompressing the wanted lang at runtime might help things out. Even a very simple LZ scheme goes a long way. We'd also need to keep all the characters in the font file, but that's just something we have to live with. Making a separate binary for each language does sound doable but it's just not sane. Keeping languages as a loadable binary file (like the fonts) might be a better choice, then each user can pick only the ones they need.

• Proper i18n goes beyond just changing the strings, the date format, decimal places, units and other stuff also change. Most importantly, when doing formatted strings the wanted parameter order can change and the "classic" printf style formatting can't cope with that, so maybe it'd be a good idea to use positional arguments with formatting ("The format can contain either numbered argument conversion specifications (that is, "%n$" and "*m$)", from https://pubs.opengroup.org/onlinepubs/9699919799/functions/printf.html).

• Regardless of how the language data is handled at the binary level, perhaps using a preprocessor written in python (we already depend on it anyways) might be a better way to get things done, dunno. The current method basically assumes you will compile GM9 with a certain fixed language, and that'll be the only lang available.

Dunno if any of these are a real problem, iirc @Epicpkmn11 has already helped with translations of GM9i and can clear up whether any of these will be real issues, or maybe mention others that can pop up.

[Epicpkmn11]

It's absolutely important you make 100% sure no english strings are changed by accident in the process and nothing breaks. For english, this will be a pure under-the-hood-change. Exceptions: If you find typos in the process (of course). Do you have anything to double check the required manual changes or even, automate them?

The strings being #defines helps a bit in that I think, you can't have a typo in a string name to make it not show up, the only way to have issues is if you use the wrong string. For making sure the right strings were used everywhere I can't really think of a simple way to automate that, but I do always make sure to look over everything again before committing.

Also, speaking names are important (which. Meaning: From the strings name, I should be able to somewhat conclude what's behind them. Maybe some scheme, which would convert the strings content to a define would make things easier. With all the strings scattered among all these source files, that may be a bit of work.

For the names how I generally do it is to just take the string but drop any less important words to try keep it a reasonable size. The printf % parameters are probably trickiest thing to make look good, what I've ended up generally doing is N for any number and either what it is (ex. PATH) or just X for strings, that makes reasonably understandable names like COPIED_N_FILES_TO_PATH or whatnot. Could also do like COPIED_LU_FILES_TO_S but that just doesn't look as good imo, I suppose it is a little more clear what the string is though.

And yeah, being understandable in the code so you're not constantly looking back and forth with the definitions is something I try to prioritize. Looking back on what I did so far I'll probably change a couple to be a bit more specific as a few are a little unclear.

---

The GM9 binary is already pretty large and adding more langs on top is only going to make it larger, so decompressing the wanted lang at runtime might help things out. Even a very simple LZ scheme goes a long way. We'd also need to keep all the characters in the font file, but that's just something we have to live with. Making a separate binary for each language does sound doable but it's just not sane. Keeping languages as a loadable binary file (like the fonts) might be a better choice, then each user can pick only the ones they need.

Since GodMode9 is already shipped in a zip and all decent guides should simply tell people to copy gm9 to the SD root I think storing them in sdmc:/gm9/languages/ or so would probably be the best solution for minimizing impact on the binary size while still bundling languages, would also make it extremely easy for testing new translations and such.

Assuming there's some way to get the system language, Japanese*, English, French, German, Italian, Spanish, Chinese (Simplified)**, Korean**, Dutch, Portuguese, Russian, and Chinese (Traditional)** at minimum should probably be included in some way, though ideally any language people are willing to translate to can be included.

*Requires a large expansion to the font or to be written using only Kana (GodMode9i does both since the 6x10 Kanji are nearly unreadable) **Requires a large expansion to the font, also I haven't even been able to find 6x10 fonts so will require a separate 8x8 font

Compression would probably help a good bit, though if they're not in the binary itself it might be worth keeping them plain text for ease of use with testing new ones and such, the files might not even exceed 32 KB (typical SD cluster size) anyways, though we'll have to see on that. Might also be worth making a revision to the fonts to LZ compress them since those do get quite large for proper CJK support, I wish I'd thought of that in the first place as I didn't really put any method into the file for making an updated version of the format 😅

It also might be a good idea to prompt the user to select their language immediately on first load (like TWiLight Menu++), since I'm pretty sure using GodMode9 is required to do a region change so that'll avoid getting too many complaints about "help my GodMode9 is stuck in Japanese" with all the people who buy cheap Japanese consoles but can't read Japanese. Also makes it easier for people who speak a language the 3DS doesn't support.

Proper i18n goes beyond just changing the strings, the date format, decimal places, units and other stuff also change. Most importantly, when doing formatted strings the wanted parameter order can change and the "classic" printf style formatting can't cope with that, so maybe it'd be a good idea to use positional arguments with formatting ("The format can contain either numbered argument conversion specifications (that is, "%n$" and "*m$)", from https://pubs.opengroup.org/onlinepubs/9699919799/functions/printf.html).

first just... wow, this is the first time I've heard about %n$ and... how have I not heard of this before, that's so useful xD. I honestly thought you had to use C++'s bloated modern formatting to get that.

The format string order isn't usually too big a deal, but I'll be sure to try to inform translators of that from now on as there are definitely situations where that'll help. I think it should be fine to just leave that to the translations as it seems you can mix normal args and positional args without issues. For date formats and such it'd probably work to either define the formats as translatable strings or simply use YYYY-MM-DD. For anything not used for file naming and such at least I'll probably try make them translatable. For decimal places I think it should be as simple as having strings for DECIMAL_SEPARATOR and THOUSANDS_SEPARATOR, though I'll have to see how GodMode9 does that.

Regardless of how the language data is handled at the binary level, perhaps using a preprocessor written in python (we already depend on it anyways) might be a better way to get things done, dunno. The current method basically assumes you will compile GM9 with a certain fixed language, and that'll be the only lang available.

How it's done right now is purely because GodMode9 doesn't seem to have any INI or similar parser at the moment so in the interest of a simple example I just put it in the code. My plan is to include just English as language.inl then read the rest from an INI or such. If there's no preferences on file type I'd probably just use INI and it's both very simple and very human readable. Probably just copy Wood's INI parser that TWiLight and such all use but adapt it to C since that's GPL 3 unless there's a better idea.

I guess the other main things I'd say need to be watched out for in my experience are:

• Avoid concatenation, never assume all languages have the same sentence structure even in the simplest cases. Colons are probably the safest thing to concatenate onto a string but even then I'd avoid it. (Not such an issue since it's not as common in C, but ex. "Copy to " + path would suck in Japanese where it should be "%sにコピー")
• Avoid "clever tricks" to reduce strings, just make multiple strings (ex. "A %s is inserted" would be a bad idea in English since if you did "SD card" it would need to be an, not a)
• Avoid re-using strings in different contexts, some simple ones like "Yes" and "No" are probably fine, but other words/short phrases may vary slightly even if they don't in English
• Either always or never make two strings for plural and non-plural, even if the plural and singular are the same in English. (ex. - and %d more... is the same in English regardless of the number, but in French it's either - et %d autre… or - et %d autres… depending on if the number is 1) If plurals are never accounted for and you just do file(s) that works too, but if you do it right for some strings best to do it for all

I'll make sure everything's good as I go through, just some things I watch for since you asked.

[d0k3]

I don't want to rush anything, just letting you know we're still interested @Epicpkmn11. Are you still working on this?

[Epicpkmn11]

oh, sorry I haven't done much more yet, I got distracted with other things and haven't come back to this...

I’ll try finish this up soon

[Epicpkmn11]

Alright, I think I've got more or less all the strings now, if anyone wants to take a look for any typos or the like now would be a good time: language.en.inl

I'll add the file reading of some sort and clean this up a bit in the coming days, it's pretty close now.

Notable changes I made to strings in my latest commit:

• GBA regions to match normal regions
• gameutil.c#540: Error: Not an NCCH file
• gameutil.c#735: Error: Not an NCSD file
• nandutil.c#338: NAND dump missesing data
• nandutil.c#352: Error: MBRTWL MBR is corrupt
• nandutil.c#511: Not an installable FIRM.

(Note: I feel like I might've changed another one or two but I can't remember which, just minor changes like this)

[Epicpkmn11]

This is now mostly functional, the main things left are a menu for selecting language and setting up some kinda translation management thing. Do y'all think I should just make a Crowdin project (ex. like GodMode9i's project) or do y'all have any other prereference in that regard? I think it's worth using some kinda site like that as it both makes it much easier for the translators and makes it easier to keep everything in sync.

---

I went with cJSON for the translation files since it's pretty lightweight (~20 KB) and JSON is a nice format – INI or something custom or so would maybe be simpler in the short term, but they don't get along as nicrly with Crowdin and it makes it harder on the translators.

Here's a build with a complete English JSON and a partial Japanese translation if anyone wants to give it a try: GodMode9-v2.1.1-9-gf77357ea-20221013220511.zip

The translations currently work the same as fonts, select one and you can set it as active or default, I'll probably leave that in for easy testing but I plan to make a menu that shows them from sdmc:/gm9/languages in a nice list based on the GM9_LANGAUGE string. There's only one real requirement to them which is that the GM9_LANGAUGE string must be in the first 0x2C0 bytes of the file, as I used that as the file format check. I'd recommend just making that the first string in the JSON.

[Epicpkmn11]

Language menu done, it'll now prompt first thing on the first load for you to pick your language and there's also an option for it in the HOME/POWER menu.

Note that this menu currently breaks if you have over 14 or so translation files, #792 fixes that.

[Minionguyjpro]

Hi, is this almost complete and can it be translated over Crowdin or something else?

[Epicpkmn11]

I believe all the code changes should be done, though I see a couple merge conflicts now that I'll look into tonight. I'm just waiting on how the leaders of the project want to handle actual translation, as per my 2nd latest message.

[Epicpkmn11]

Merged upstream, so it's no longer conflicting

[Minionguyjpro]

Crowdin is a good site to put this, I also contributed to the DS-Homebrew translations.

[Epicpkmn11]

A Crowdin project is now mostly set up, just needs more screenshots/context descriptions in general: https://crowdin.com/project/GodMode9

I copied the languages GodMode9i's Crowdin has for now (with the exception of RTL languages), just let me know on Crowdin/Discord/IRC if anyone needs a language added and don't hesitate to comment if anything needs more context

Testing translations is really easy, just download, put on your SD, and choose to set it as your language

[Minionguyjpro]

A Crowdin project is now mostly set up, just needs more screenshots/context descriptions in general: https://crowdin.com/project/GodMode9

I copied the languages GodMode9i's Crowdin has for now (with the exception of RTL languages), just let me know on Crowdin/Discord/IRC if anyone needs a language added and don't hesitate to comment if anything needs more context

Testing translations is really easy, just download, put on your SD, and choose to set it as your language

Nice. Will look into it.

[Minionguyjpro]

When will these changes be merged and added into the application?

[d0k3]

Alright, I'm sorry it took me so long to get back. Real life has it ways of sabotaging your personal projects, and that's what's happening when I'm taking months for reviewing a change. I got to say, though, this is the largest pull request GM9 ever had.

That being sad, this does look very good. @Epicpkmn11 - thanks for this one!

I'm ready to merge this. @Wolfvak @aspargas2 - want to have a look over this pull request, too, before I do so?

[aspargas2]

So I haven't been following with this PR much, but I'll try to give my thoughts on it as best as possible anyway.

First off, I am a bit apprehensive about the global addition of -Wno-format-nonliteral; it seems like it should be possible to make use of some preprocessor magic involving #pragma GCC diagnostic to only suppress the warning when using a translation string. If a way to do this is found, the changes in commit 5da0218 wouldn't be necessary either. I'll have to think on that for a bit, but if no other issues are found, I'd say go ahead and merge it as is and I'll worry about that later.

Also, is there any plan to deal with the fact that the only current translation (Japanese) is basically broken without manually loading a custom font? Maybe it would make sense to have a system to bundle an alternate default font to use with certain languages in cases where it is necessary.

[Minionguyjpro]

Nice. Waiting on it! It is nice to be able to have every homebrew application on your 3DS system in your own language!

[Epicpkmn11]

Also, is there any plan to deal with the fact that the only current translation (Japanese) is basically broken without manually loading a custom font? Maybe it would make sense to have a system to bundle an alternate default font to use with certain languages in cases where it is necessary.

Oh, tbh I forgot that was still included since I've been intending to open up a second PR actually adding the languages people have done a decent amount of once I've got the time. I've not had much free time lately, but that should be changing soon so I'll try get that started soon.

Also, as discussed a bit on the Discord I'll take a look at using a more efficient custom format than JSON to cut down on unneeded code and remove all the strdups.

[Epicpkmn11]

I've switched from JSON to TRF (Translation RIFF, custom format), fixed a couple minor bugs that I noticed, and added support for per-language fonts (when loaded from the language menu it will check for a file named the same as the language but with the .frf extension instead of .trf, if there is it'll load that and set it as default too)

I've also opened #802 adding the actual translation files and a Japanese font (GodMode9i's, but with the Hebrew and Arabic removed, as GM9 has no RTL support) using the aforementioned font loading feature

[Epicpkmn11]

Just updated the README with a section on fonts and translations, not sure if this is the best place for it but it seemed fitting?

I also added myself to the credits (I hope that's alright), since it's not sorted wasn't sure exactly where so I put myself below the previous latest change (if I read the blames right)

[Epicpkmn11]

Very glad I decided to finally stop being lazy and finish translating to Japanese (about 70% now), I've noticed several fairly minor non-English affecting bugs that can affect translations and I'd otherwise missed

Mainly related to missing UTF_BUFFER_BYTESIZE on some arrays/snprintfs, and a few non-ideal strings

[d0k3]

Thanks a lot, @Epicpkmn11 ! Does that mean all issues we mentioned earlier are solved? Maybe @aspargas2 and @Wolfvak will want to take a look at those recent changes, too. I'll make sure to have a thorough look later in the weekend.

[Epicpkmn11]

I just realized I missed the date/time format and decimal/thousand separators despite having been explicitly mentioned here 😅, but just did that so I believe everything should be addressed apart from @aspargas2's pre-processor magic suggestion as I'm not sure how you'd go about doing that tbh

I think everything should be stable and good to merge, though please let me know if you find any issues. I've done my best to test and checked a decent amount of places while doing the Japanese translation, but I haven't tested everything as a lot of strings are rather tricky to get to naturally

[Epicpkmn11]

I put together a build merging this and #802 for testing convenience, attached below

GodMode9-v2.1.1-37-g961d0e27-20230323184608.zip

[aspargas2]

Alright, sorry for taking so long to get back on this. I've just looked over most of the diff and noticed some things I'd like to mention.

• In commit 4d3d429, I think you accidentally removed a call to DrawStringF entirely and didn't replace it with anything, on line 251 of godmode.c. It would probably would never have been missed since it's under an ifdef that isn't ever used by anyone to the best of my knowledge, but it should be fixed nonetheless.
• There are some improvements that could definitely be made to the new code in language.h and .c. A struct definition or two should probably be created for the custom language file format instead of doing lots of manual pointer manipulation and memcpy and memcmp, and while I admit that code style throughout the repo is a bit inconsistent at times, compLanguage should probably be renamed CompLanguage to at least be consistent with the other functions in the file. I'd be happy to work on improving the parsing code myself if you'd like.
• A hypothetical malicious language file could easily pwn the whole system using a format string exploit, as translation strings are used as format strings in many places. I don't see any practical way to avoid this without overhauling how we're handling string formatting in GodMode9 as a whole, so it's just something we should be aware of

Overall, once the first item there is fixed, I'd say this PR is probably fine to merge. Improving the parsing code, aforementioned preprocessor nonsense to deal with suppressed warnings, and some improvable spots in pre-existing code I noticed but didn't mention here can be taken care of later.

[Epicpkmn11]

Been discussing on Discord, summary:

• This was an intentional change since for me it was instant and just caused a slight flicker, however it seems it does sometimes actually take a long time so I've undone this (to a fixed version)
• I've improved the clarity of the RIFF parsing code on both the language files and the existing font RIFF parsing so there's less magic numbers and pointer magic going on, removed some pointless copies too. I've also cleaned up the language.c formatting a bit for better consistency
• Yeah, definitely good to be aware of it, but given what GodMode9 is I don't think it's enough of a concern to be worth doing anything about

[aspargas2]

After the aforementioned discussion on Discord and last few commits, I'd say this is ready to merge.

cc @d0k3 @Wolfvak

[d0k3]

Alright, looks good to me, too. However, could you rebase your commits so that I can merge without a merge commit?

[Epicpkmn11]

Rebased on master, if you'd like I can also condense some of the minor commits

Edit: See translatable-squashed

[d0k3]

If you don't mind, some condensing would be nice. And, thank you a lot for being as quick as you are!

[Epicpkmn11]

I've condensed it all down to five major commits, I kept a copy of the previous history so I can confirm it's identical

[Nemris]

If it's wanted, I could refactor the transriff.py for modularity and to reduce file writes.

[d0k3]

Merged! Thanks a lot, @Epicpkmn11, for your great work and also for your patience. Now, I guess we'll have to merge that other pr.

@Nemris - any kind of improvement is highly appreciated. Feel free to open a pull request for this.

[Nemris]

I already have the changes ready, @d0k3. As soon as I get some time, I'll rebase on top of master and send a PR.