Localization or different default language

[jongleur1983]

Hi. I'm quite interested in this game, looks nice so far - but I don't speak Spanish, so the not only default, but only option to get a spanish in-game user interface is a showstopper for me.

Therefore I propose to add multi-language-support (and I hope it get's interested enough to keep me on track to participate). I could contribute German (native speaker) and if nobody else could, English.

[OpenHogwarts]

We would like to have multilanguage, however we don't have experience in this area and looks like Unity is not ready for this by default, so it may take a while to do it :(

Spanish community was the only that participated in the development rather than expecting someone else to do it all, so the default lang is spanish.

[jongleur1983]

Hi. Thanks for the quick response. I didn't use Unity3D before (although I saw it in use already), I'm more a C# developer.

To get a first step towards it I'm currently trying to replace the setup method of the database as follows: currently the item data is stored in static C# source code. Instead I would like to introduce an XML file that contains the data and is read from the database initialization code.

In a first step this would allow to select one of many languages at compile time, keeping the database schema as it is now. On top of that it would be quite easy to a) use a language fallback if a string is missing and b) to add more languages by just editing an xml file instead of code.

So far this would not add multi language support at runtime, as I would have to understand more of the database for that before I can try it.

I hope to get a Pull Request ready until tomorrow night that adds these capabilities without adding any additional features - if not it'll be not before in two weeks unfortunately as I'm quite busy in the next days.

[OpenHogwarts]

Cool, i don't like to have that data embedded/hardcoded in the code . For lang translation, we will probably need to extend or overwrite UnityEngine.UI.Text class. That class is responsible of displaying almost all texts in UI:

With that, we could then build a key (default lang) => lang JSON files. en.json

{
"Guardar": "Save",
"Aceptar misión": "Accept mission",
}

de.json

{
"Guardar": "Speichern",
"Aceptar misión": "Akzeptieren mission",
}

So you can serve the requested key to translate as default if no translation is available. It would be fast to implement as it wouldn't need to replace all existing labels.

[jongleur1983]

From my experience with previous localization issues (a bigger one in a C#/WPF project and something in Wordpress and PHP) I would not use keys as readable fallback values but some way of artificial key that's more describing (keep in mind that words might require context to be translated correctly, and a single word that might be used twice in Spanish might be better translated to two different ones in another language). Nevertheless it would be possible to define e.g. Spanish or English as the fallback language to use.

Nevertheless I'm starting with the database for now as that's C# only and I'm more experieced there yet.

[cal97g]

XML is aids, it should be a JSON file.

[jongleur1983]

as a programmer and as a general statement I disagree: JSON is ugly to parse, the only benefit is it's smaller file size, something that doesn't matter for games at all (in the scale of text files in code that's compiled to something different. XML is better supported by tools once you want to have some kind of Schema we could introduce here.

Nevertheless I'm the newcomer in this project, so I'm fine with whatever is decided, it's not a big difference.

Independent of the project and this issue I would be happy to get your arguments against XML - and in which way JSON fit's better (in general - as your statement sounds to have that scope, and in this particular issue). Thanks.

[OpenHogwarts]

As webdev, JSON is the obvious choice (for web) because of as you said, it's size.

For this game, size doesn't matter at all, it only matters how easily is to work with JSON or XML when coding.

• JSON: Unity has JsonUtility class (https://docs.unity3d.com/ScriptReference/JsonUtility.FromJson.html) but it would require to have classes like ItemData set as [Serializable] which i don't know if it's possible when they have methods, and what would happen to IBoxDB having that set.
• XML: I can't find references in Unity's Documentation (https://docs.unity3d.com/ScriptReference/30_search.html?q=xml) maybe it's already easy to work with C# by default. (With C# I've only made simple games in Unity, so i'm not an expert/normal dev)

So the question is, ¿Is it easy to work with in C#?

@cal97g could you "extend" your arguments?

[jongleur1983]

in web you deal with javascript, and JSON is - in it's core, just javascript. In C# XML parsing support is quite good, JSON-Support exists as well.

As I said: it doesn't matter, and if you think that JSON is the better way to go, I'm fine. From a programming perspective outside unity itself (that's my starting point yet, as I'm going to put the data into IBoxDB as ItemData objects for now) it doesn't matter.

After a short look into the SimpleJson.cs (included in the project sources) this seems to be as easy - except that it has to be done by hand as well, as long as the database schema isn't capable to take multiple languages at the same time.

Nevertheless: I fear I'm not going to push anything this and the next week as I didn't had time today and I'm busy the next week, but hope to continue afterwards.

[jongleur1983]

okay, still struggling with loading the resource file from DBSetup.cs, no idea what I'm doing wrong here. Won't fix it before next weeks friday, but thanks for your support so far ;)

[OpenHogwarts]

You can push the code on a branch/fork, so we can take a look and maybe find out what is happening :)

[cal97g]

Json is generally easier to work with and better supported in my experience. Plus, it looks nicer subjectively and anybody could edit it. I've known XML be difficult to work with. I suppose it would depend on the tools available. Json is easier to read and edit.

[tetreum]

I had to make a translation system that includes the UI text component replacement too for another game. Maybe i apply it in this game/ i release the system in a repo. Uses json for the phrases.

[kardall]

I forgot to put the note in the last push...

I added a Locale class. It has the ability to 'Add and Retrieve' Quest Text. So it'll be up to the person adding the Quests to read on how it works to actually use it.

For the most part, the actual adding of quests is the same, it's just that, when you use the actual Title/Pre/After parts, you can call the Locales.AddQuestData() and Locales.GetQuestData() functions based on the language (Yes there is a checkbox on the front login screen for English Text and it is used in there as well.

There is also a convert PlayerPrefs.GetString("Locale") converter so you can use it in the quests.

Essentially, you send a request to add or retrieve the data by QuestID and the Locale of the player with:

Locale.GetQuestData(quest.questid, Locale.ConvertPlayerLocale());

Or something like that. I haven't actually tested adding / retrieving quests cause I can't translate LOL But all the code is there and it SHOULD work. You just have to work on the quest side of it and how to add it. I haven't gotten that far yet. But the beginnings are there.

I was going to make a specific class for other things like items and regular text. So we could have text on the main screen in english :)

[tetreum]

I forgot to release my system, i will leave here the code, it can handle the UI/any text translation:

Assets/Plugins/LanguageManager.cs

using System;
using System.Collections.Generic;
using UnityEngine;

/**
 * ¿Why is LanguageManager placed in /Plugins/ folder?
 * https://docs.unity3d.com/Manual/ScriptCompileOrderFolders.html
 * To give him compilation priority over any other script (so translation is loaded before anything else)
**/

public class LanguageManager : MonoBehaviour {

    public static Dictionary<string, string> translation = new Dictionary<string, string>();
    public static Dictionary<string, string> fallbackTranslation = new Dictionary<string, string>();

    private static SystemLanguage fallbackLanguage = SystemLanguage.English;
    public static SystemLanguage[] availableLanguages = {
        SystemLanguage.Spanish,
        SystemLanguage.English,
    };

    public static SystemLanguage? _playerLanguage = null;
    public static SystemLanguage playerLanguage {
        get {
            if (_playerLanguage == null) {
                if (!PlayerPrefs.HasKey("Language")) {
                    PlayerPrefs.SetString("Language", Application.systemLanguage.ToString());
                }
                _playerLanguage = stringToSystemLanguage(PlayerPrefs.GetString("Language"));
            }
            return (SystemLanguage)_playerLanguage;
        }
        set
        {
            if (System.Array.IndexOf(availableLanguages, value) == -1) {
                Debug.LogError("Attempt to set invalid unavailable language: " + value.ToString());
                return;
            }

            PlayerPrefs.SetString("Language", value.ToString());
            _playerLanguage = value;

            reloadTranslations(true);
        }
    }

    void Awake ()
    {
        if (translation.Count == 0) {
            reloadTranslations();
        }
    }

    public static void reloadTranslations(bool reloadUI = false)
    {
        translation = new Dictionary<string, string>();
        fallbackTranslation = new Dictionary<string, string>();

        if (System.Array.IndexOf(availableLanguages, playerLanguage) != -1) {
            getTranslations(playerLanguage);
        }

        if (Application.systemLanguage != fallbackLanguage) {
            getTranslations(fallbackLanguage, true);
        }

        if (reloadUI) {
            TextI18n[] translatableTexts = GameObject.FindObjectsOfType<TextI18n>();

            foreach (TextI18n text in translatableTexts) {
                text.Refresh();
            }
        }
    }

    public static SystemLanguage stringToSystemLanguage(string language)
    {
        return (SystemLanguage)Enum.Parse(typeof(SystemLanguage), language, true);
    }

    public static string get (string key)
    {
#if UNITY_EDITOR
        if (translation.Count == 0) {
           reloadTranslations();
        }
#endif
        if (translation.ContainsKey(key)) {
            return translation[key];
        }

        if (translation.ContainsKey(key)) {
            return fallbackTranslation[key];
        }

#if UNITY_EDITOR
        PhraseTranslation phrase = new PhraseTranslation();
        phrase.key = key;
        phrase.translation = key;
        Debug.Log("NOT TRANSLATED:\n" + JsonUtility.ToJson(phrase, true) + ",\n");
#endif
        return key;
    }

    private static void getTranslations (SystemLanguage language, bool fallback = false)
    {       
        TranslationsFile file = JsonUtility.FromJson<TranslationsFile>("{\"translations\":" + ((TextAsset)Resources.Load("i18n/" + language.ToString())).text.TrimEnd('\r', '\n') + "}");

        if (file.translations == null) {
            Debug.LogError("Translation file for " + language.ToString() + " is invalid");
            return;
        }

        foreach (PhraseTranslation phrase in file.translations)
        {
            try
            {
                if (fallback) {
                    fallbackTranslation.Add(phrase.key, phrase.translation);
                } else {
                    translation.Add(phrase.key, phrase.translation);
                }
            } catch (Exception) {
                Debug.LogError("Duplicated key " + phrase.key + " for " + language.ToString());
            }
        }
    }
}

[System.Serializable]
public class TranslationsFile
{
    public PhraseTranslation[] translations;
}

[System.Serializable]
public class PhraseTranslation
{
    public string key;
    public string translation;
}

Assets/Plugins/TextI18n.cs (This component replaces unity's default Text component)

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;

[AddComponentMenu("UI/TextI18n", 10)]
public class TextI18n : Text
{
    void Awake() {
        if (color == Color.white) {
            color = new Color32(50, 50, 50, 255);
        }

        Refresh();
    }

    public void Refresh() {
        text = LanguageManager.get(text);
    }
}

And finally, create a json file for each language you want. Example: Resources/i18n/English.json:

[
    {
        "key": "Crear personaje",
        "translation": "Create character"
    },
    {
        "key": "Empezar",
        "translation": "Start"
    },
]

You don't need to place all phrases at once, as system will yield you by console any phrase that hasn't it's translation set.

To get the translated phrase just call LanguageManager.get("Crear personaje");, if available will return the translation, otherwise will return the given string.

It is recommended to make the game in any other language than english, so you can easily see which parts are not begin translated when testing it. You're already making it in spanish by default so it's perfect for this case.

[kardall]

Seems interesting :) I didn't go 'file based' just cause I wasn't sure about where this was being store, and whether or not it would have database requirements later on... not sure ;s

[tetreum]

Well, for translations usually its usefull as you can give them to nonprogrammers for translation, and there are apps that make it easy for json files :p

[jongleur1983]

Cool to see progress here - and shame on me I didn't progress earlier (but failed with unity for now). Nevertheless I'd like to add something to the argument @tetreum mentioned:

It is recommended to make the game in any other language than english, so you can easily see which parts are not begin translated when testing it. You're already making it in spanish by default so it's perfect for this case. I disagree, but would add a strategy tip for it for different reasons: Don't use the real complete text as a key as it should occur in the UI in any given language.

The sample @tetreum gave in his post uses spanish keys, so he's right: With this it's not possible to distinguish between the Key you get as a fallback and an existing translation.

Instead if artificial keys are used, this can be made in a way it's simple to distinguish. E.g.: Make keys entirely upper case. If you see a key in the UI it's not available in the given language (for debugging, switch off fallback language, after debugging, switch it on again to get reasonable fallbacks in the wrong language at least).

A second argument to use artificial keys that are not exactly the visible text: Whatever language you decide to use in the key, you may not be able to express the complete, non-ambiguous meaning of the text item. Let's take the English text "it's raining cats and dogs", a common proverb on the one hand, but in hogwards it could be literally that due to some curse that happend before. If we translate it to German under the estimation it's the proverb, we would use e.g. "Es regnet Bindfäden" (roughly "it's raining twines", translated back word by word), something never expected here. So the key might better be chosen "ITS RAINING CATS AND DOGS (PROVERB)" to tell anyone doing the translation what's the purpose of the text.

A third argument is, that language is always context aware. The same sentence in English used in two different situations may be translated differently to Spanish or German, depending on the context it's used, be it different emotions or a different kind of relationship between characters. Take anything including the pronoun "you" in English, that may translate to "Du" (formal, used between friends) or "Sie" (more formal, used in official language and between foreign adults). Without context any sentence including "You" in an English original can't be translated properly.

[OpenHogwarts]

But that happens with key approach too, because what matters is the language begin used in those two cases. If in english you use the same word for 2 things that may have diff words in a another language, your key will be the same, until someone tells you that it must be different so you change "KEY_BACK" -> "KEY_BACK_2". With phrases it's the same, oh diff translation needed? Let me write a new phrase. "Back" -> "Exit" or "Back" -> "Go back"

Right?

[jongleur1983]

partly right. You may end up with different keys that have the same content in one or more languages. You're right: A developer coding in his mother tongue might re-use the same key as he's sure it's the same content, and some translator realizes that's not the case as in his (target) language it's different. In this case this may lead to an issue being raised "we have to split the i18n resource X as it's content differs in some languages between use case a and b", but that's fine. While splitting this, the keys then can be refined to "x_when_used_in_case_a" and "x_when_used_in_case_b" (of course described by a better name).

Using the wording itself in the key and relying on that disallows explanations for translators or developers.

[themiliton]

Did the method of implementation ever get decided? I am more than happy to work on typing on the English translations that would be required if there is a some way of creating this.

[OpenHogwarts]

@themiliton it got started in https://github.com/OpenHogwarts/hogwarts/pull/24 but got frozen

[themiliton]

Ah well. I'll have to find another way to help. I'll carry on working my way through Unity tutorials and learning more about how it works until I get to a point where I can be more useful!

[OpenHogwarts]

Localization support, based on what @jongleur1983 said, released https://github.com/OpenHogwarts/hogwarts/commit/7a2803c672e5fc55eb8e5947922ae88169deb2cf

The game is now available in English (not 100%) & Spanish. Language selection will be based on your Windows lang.

Localization wiki: https://github.com/OpenHogwarts/hogwarts/wiki/Localization

[jongleur1983]

Thanks for getting this into place.