Better Multi-Language Support

UTDZac commented 1 year ago

This PR offers a way for the Tracker to become fully translated in languages other than English. To accomplish this, anywhere and everywhere that text appears on the Tracker has been "extracted" and moved to a resource file for it's corresponding language.

The ultimate goal of this project is to get help from the community to translate the Tracker into different languages. This will be accomplished through a GitHub Discussion forum.

This PR is massive beyond reason and I don't expect a full review of everything. I would like some extra eyes on some of the larger-scale changes.

Features

99.9% of all Tracker text that is displayed is now stored in their respective language file. Over 700 entries.
- For example, All English resources can be found in /Languages/English.lua
A new Language settings screen is available.
- The display language of the Tracker can be changed independently of the language of the game rom being played (mostly, see known issues)
- An option to "Autodetect language from game" is on by default, which will update the Tracker's display language based on the game being loaded.
A new Release Notes page is available, which can be shown/hidden from the Update settings screen.
And maybe some other things I probably forgot. Been working on this one for a while.

Primary Files to Focus Review

Resources.lua
- Change DEBUG_REPLACE_ALL_RESOURCES to true if you want a quick way to scan through the Tracker to make sure text resources are working and get replaced. Some names of things might not render or function properly with this enabled.
- When you're done testing, let me know and I will remove the debug code, or you can do it.
Languages/English.lua
data/RandomizerLog.lua (regex changes)
screens/LanguageScreen.lua
screens/UpdateScreen.lua

Questions

Q1. Should I copy/paste all English.lua entries for all other Lua language files? I originally left them out, since there aren't any translations available, but perhaps it's easier to have all those text-keys already there instead of requiring others to add them in themselves. Granted, it will be us who eventually will add them all in, but if someone wants to translate on their own without our help, it might be easier if those entries were already there.
A1. I went with this solution. All entries from English.lua have been copied over to the other language lua files.
Q2. With regards to getting help from the community on translating the other languages, I'm unsure on the best way to present this info that will be translated. I was thinking having it split by priorities might be the best way to go? "Highest Priority" being the most common visible stuff, like the main tracker screen and settings, "Mid Priority" being other menus like Log Viewer, and "Lowest Priority" being rarely used texts such as move/ability info descriptions or form prompts.
A2. I'll be using the priority list system. Initially I'll just have the high-priority language entries that need translation online first. After some time, I'll add the other priority entries.

Notes & Known Issues

Several MGBA resources are duplicates. This is intentional such that a translator can simply translate just the MGBA resources if they want to use those, and don't have to go hunting for the same resource hidden elsewhere.
Some texts and resources are not translated. Usually this is because they are error messages that might appear before the Resources lua is loaded, or because I felt they were minor and not worth the time. Let me know if you think otherwise.
[FIXED] If the display language of the Tracker doesn't match the game rom language, then the log viewer won't function properly, since it has to match the names of things in the log with the names of things in the Tracker.
[FIXED] Bizhawk 2.8 encodes accented characters, and as such those won't properly be uppercase/lowercase in some places. This isn't an issue with Bizhawk 2.9 or MGBA.
[FIXED] The LogViewer search screen currently doesn't work for accented characters. I couldn't find a clean solution for this, see Utils.containsText()
While the Japanese language isn't official supported, the Tracker will allow loading a Japanese FireRed rom and playing it just fine.
[DEBUG REMOVED] Please don't merge this PR in until the DEBUG stuff has been removed. It's only there to help with testing and reviewing for this PR.

Fellshadow commented 1 year ago

Also mentioning #62 as this is related and that issue is sorta our main "non-english support" update/progress log

UTDZac commented 1 year ago

Fixed the ability to search for names of things containing accented characters in https://github.com/besteon/Ironmon-Tracker/pull/390/commits/6f81cd6968a80c402a611ac4a95901930b8f3549 , using the keyboard that only has plain text characters. This required a good bit of reorganization work to get it done, partly why I avoided it in the first place, but had the energy today to get it resolved.

One lingering issue that was present before and after this change is that on Bizhawk 2.8, for names of things that start with special characters, they don't seem to render properly... and I'm not entirely sure why.

ÜBERWURF (Vital Throw) is not displayed correctly.

Fellshadow commented 1 year ago

Fixed the ability to search for names of things containing accented characters in 6f81cd6 , using the keyboard that only has plain text characters. This required a good bit of reorganization work to get it done, partly why I avoided it in the first place, but had the energy today to get it resolved.

One lingering issue that was present before and after this change is that on Bizhawk 2.8, for names of things that start with special characters, they don't seem to render properly... and I'm not entirely sure why.

ÜBERWURF (Vital Throw) is not displayed correctly.

@UTDZac Had a little dig on my bizhawk 2.8 install, seems it's coming from the RandomizerLog.setupMappings somehow failing to properly format the move name as lowercase (resulting in a failed move to id mapping later), which is weird because the same toLowerUTF8 and formatSpecialCharacters Utils functions work fine when parsing the log?

Attaching some Console output below, the umlaut characters print to the console weird but as you can see they get formatted fine in the end. Although this then makes me wonder if it's down to a difference in character encoding between our own files being read by RandomizerLog.setupMappings and the log file being read by e.g. RandomizerLog.parseLog and it just didn't affect the other characters because they happened to be lowercase to begin with, though that would be weird considering we have the characters defined the same way in Constants.Chars 🤔

UTDZac commented 1 year ago

Fixed the Bizhawk 2.8 encoding issue in https://github.com/besteon/Ironmon-Tracker/pull/390/commits/4f892875c3e0ba3705e66c620f43726da22860bb by just adding duplicate entries to Constants.Char for each of the special characters that require encoding.

For example, instead of just one entry for ý there are now two:

    ["ý"] = {
        encoded = "\253",
        upper = "Ý",
        plain = "y",
        width = 3,
    },
    ["\253"] = {
        upper = "\221",
        plain = "y",
        width = 3,
    },

UTDZac commented 1 year ago

https://github.com/besteon/Ironmon-Tracker/pull/390/commits/2aee88c1b1a46029596afe9b7d1366a04a5fe888 Fixed the issue where the log couldnt be parsed if the Tracker's display language didnt match.

UTDZac commented 1 year ago

Fixed a few more issues relating to searching for names of things in the Log Viewer Search.
The Log Viewer should now reflect language changes when the setting is changed. Works in most cases I could find.
Added placeholder language resource entries for all non-English languages, noted by a trailing -- NEEDS TRANSLATION. It's not pretty, but it gets the job done.
Removed old, redundant move and ability descriptions from their respective data files. Those descriptions now live in their respective language resource files
Removed all references to Constants.Words.POKEMON and .POKE.

With all that said, I'm ready to merge this in. Just wanting to see if you have any final thoughts.

besteon / Ironmon-Tracker