Multi-language/I18n support

[proddy]

Support multi-locale for Telnet console commands, MQTT, SysLog messages and the Web. Since there is not enough memory to hold all the translations it will have to be different builds (DE, NL, UK). String literals will be stored in translation files and built at compile time.

@MichaelDvP @bbqkees what you vote for this feature?

[bbqkees]

Most Gateways go to German speaking countries so it makes perfect sense to provide the interface in their own language. I get some questions about it regularly. If there is not enough memory to store everything, we should at least provide a language selector at the first install which will then download and install the new firmware. So we could just have the UK build as default with a selector to install another language.

[MichaelDvP]

Yes, that would be nice. What about a changeable language-file in the spiffs? There is a lot of space left and you have only one software.

[proddy]

That's a good idea. Store the localizations in SPIFFS and download to PROGMEM when EMS-ESP boots.

[proddy]

In order of importance

• Web UI
• MQTT topics and payload
• Console commands (serial and telnet)
• Logging information (to console & syslog)

For the web I had a look at react-intl, i18next-react, polyglot and this one called lingui-js but they'll all quite big libraries and there is just not enough progmem to store all the translations to have it real-time on the ESP8266. react-intl is the right one here (14k zipped).

[proddy]

First localized version of the UI is looking promising.

[proddy]

with the extra 160K on the ESP32 I can easily store the strings in SPIFFS and load them dynamically

[proddy]

Going to look at this again.

[proddy]

started working on this again. It's easier now since we moved React to use hooks. I have selectable languages for EN and DE and starting to translate the UI. I've been thinking how to store the translations and ideally, it would be great if this was in a custom JSON file the user can upload and replace, but for now I'm hardcoding it.

@MichaelDvP one thing that struct me is the amount of memory we have left on the ESP32. I remember the countless long weeks I used to spend optimizing every heap byte on the ESP8266 with version 2 so it wouldn't run out of memory and crash. On the ESP32 we have still 250KB of RAM and Flash both available so I was wondering what difference it would make if we ran the web pages not in PROGMEM and also all the F_() text literals in normal Memory instead of Flash. It may be faster ! I'll run some tests and see.....

[proddy]

The web code is 300KB so needs to stay in flash, but its cached by the browser anyway.

[proddy]

I finished the coding on this. You can switch real-time between UK and DE and the Web UI and Device Entities change. Problem is I've run out of Flash memory so now need to go back to the drawing board to see how to squeeze this into the tiny 4MB Flash the E32 and S32 boards have. On BBQKees's latest prototype it works like a dream since it has an embedded ESP32 chip with 16MB flash!

@MichaelDvP I might need your expertise here to hack the IDF partition tables.

[proddy]

I managed to squeeze in the languages into the 4MB flash and make it dynamic. So you can adjust the language in the WebUI from the sign-on screen and also change the language of the device entity names from the Settings without re-starting the ESP32. A few notes

• I've done German (DE) for starters. It's most likely all wrong as I used google translate, sorry
• The WebUI language translations are in interface/src/i18n/de and interface/src/i18n/en. The English (en) language is leading and any other language file should match. To create new languages simply add a new folder.
• Translations to the web can be added and tested real-time by running npm run standalone from the interface directory. No need to upload to the ESP32.
• I've only translated the login screen and the Dashboard so far in the web. The rest is easy but time-consuming. I'll add bit by bit.
• The languages in the EMS-ESP core are stored in system.cpp as const char * const languages[] = {EMSESP_LOCALE_EN, EMSESP_LOCALE_DE};. This can be expanded but the sequence is important. So 0 for EN, 1 for DE, etc...
• The core files are located in a single file called src/locale_translations.h. I've missed some of the German translations and have marked these with a comment // need DE.
• I made a few fixes and optimizations in the code as I re-looked into code again. It's nice to be back! For example the way we store device entities and manage the numerical operations like div2, div10 etc. This does mean I've probably introduced a few more bugs but we'll fix along the way just like old times
• When we have DE done we can think of adding Dutch NL and Swedish SE. I think we can safely add up to 5 more languages without blowing the memory, If we do, we'll have to fix the language at compile time,

The code is at https://github.com/emsesp/EMS-ESP32/tree/v3.5.0

@bbqkees @MichaelDvP can you give it a whirl and see if it works?

[bbqkees]

Ok great work. Everyone will love this feature. I'll try out it in the coming days.

[MichaelDvP]

Wow, great work. Amazing that it fits to 4M. Tasmota have introduced a "safeboot" shortly. That's (as far as i understand) a smaller OTA_0 partition, holding only a minimal image with OTA upload and a larger OTA_1 for the firmware without upload option, only a switch to reboot to safeboot partition. Maybe that's a way.

Fist look, i had difficulties to find the web-language setting, maybe this could help: and for language selection don't use the translation, better name the field with all languages to find your own (if set to chinese i have difficulties to identify the language field on some websites):

[proddy]

Good points for the WebUI. I'll change that. I also tried adding the language on the top bar (next to the person icon) but it messed up with the alignment. It should be possible though if you think its helpful

On OTA, I thought the partitions must be the same size (ota-1 and ota-2). The first holds the app and the flash memory and the 2nd is used for uploading over OTA. I used this site as a guideline https://blog.espressif.com/how-to-use-custom-partition-tables-on-esp32-69c0f3fa89c8

[MichaelDvP]

Yes, normal scheme is two identical OTA partitions, but read this: https://tasmota.github.io/docs/Safeboot/

[proddy]

wow, that's cool. Using the factory partition is clever (https://github.com/arendst/Tasmota/blob/development/partitions/esp32_partition_app2880k_fs320k.csv). We could make a smaller firmware image without the web

[proddy]

I added German translations to the Settings, Customizations and Help Web screens. I'm sure its wrong and funny, but it's easy to modify. Next I'll work on the System scenes. My god, this is boring!

[MichaelDvP]

Wasn't it easier first to add NL and when all tags are in, leave the DE to native german? I'll check the DE next week. First i'll complete the locale_translations.h

BTW: i've tested a bit more. Thermostat set_mode does not work in DE, in set_mode and mode_tostring it should set the to-lower flag, or, as mode_tostring is only used for set_mode, we can skip it and modify set_mode to:

        case EMSdevice::EMS_DEVICE_FLAG_RC10:
            mode = Helpers::translated_word(FL_(enum_mode6)[num], true);
...
    if (Helpers::translated_word(FL_(off), true) == mode) {
        return set_mode_n(HeatingCircuit::Mode::OFF, hc_num);
...

I can do a PR after you finished your changes.

Is it usefull to translate the mqtt output? The shortnames are still english. Changing enum/bool is easy, but strings like ' switchtime` are set in device and will be the same in web/mqtt/api. But it's only very few strings and i think you have a new idea for holidaymodes and switchtimes,which are now handled by the combined strings.

Is it possible to translate the tags for tag_to_string . In german we should use dhw->WW (Warmwasser), hc->HK (Heizkreis), hs->WE (Wärmeerzeuger, according to buderus manual), but only for web output, for mqtt/api (tag_to_mqtt) i think it's not usefull.

[proddy]

I did DE as you had most of the entities already translated. The rest was done automatically using a script in VSC querying Google translate. I'm not Dutch so would also have struggled with the NL translations. I'll continue with the remaining web pages and leave it all in EN.

translating the shortnames will be more code and less memory but it is possible.

translating tag_to_string should be easy.

[MichaelDvP]

translating the shortnames will be more code and less memory but it is possible.

I mean leave shortnames in english and don't translate values for mqtt output. Not sure about api, because api can check the value info to get right options for an enum.

Also commands are shortnames, i think they should stay and use hc tags.

[proddy]

@MichaelDvP I've completed localizing the WebUI, all except the validators which I'll look into later. The DE translations for these are in interface/src/i18n/de/index.ts and prefixed with "DE_".

I'll be out for the next weeks so will have little time for fixes so please go ahead and hack away and fix those set_mode bugs. When we have DE working we can push 3.4.2 to stable/main as a bug fixing minor version and then this 3.5.0 to dev. Then @bbqkees can work on the NL translations!

[bbqkees]

Yes it's better for me to wait until DE is completely ready so we have all items completely in the same files to be translated.

This localization also solves the friendly names issue in Home Assistant. Lot's of people translate the friendly names into their own language. If you give an entity a new friendly name in HA, and EMS-ESP is rebooted, HA will overwrite the new friendly name with the old one.

[proddy]

Next feature I'll add, which is relatively easy, is to rename the entity names. From the customization web page you can click on an entity name and a dialog window is shown where you can change the 'fullname' or reset it to the original.

[MichaelDvP]

I've updated the german translation and pushed direct to v3.5.0. I'm not sure about some wordings. Dashboard (Armaturenbrett/Instrumententafel) is used in german only for cars. I named it Kontrollzentrum (control center). Words like Access Point, Upload, Download are also common use in germany, not sure if they should be translated.

For RC300, etc. maybe a german owner can check. @tp1de Do you like to check the translations and make suggestions? You have to download and compile the v3.5.0 branch.

I think it will need some more iterations, i always find something that needs better wording. But we are in the beginning of v3.5. beta.

[bbqkees]

Ok great can I start with the NL translations now?

[MichaelDvP]

I think yes. Have you found all places to add the new language? system.cpp line 45 i18n-util.sync.ts, line 8-14 the web translations go to new i18n/nl/index.ts entitiy translation in locale_translations.h

[proddy]

not i18n-util.sync.ts as its auto-generated during the npm build webpack.

You will need to add 'Nederlands' to /interface/src/project/SettingsApplication.tsx for the language setting option.

Maybe it's just better if we create the NL stub and you just modify the files?

One thing I want to go back and re-do is the plural's - so using a function to work out the difference between Sensor/Sensors Sensor/Sensoren etc.. without having to do a translation for each. It's built into the i18n library (https://github.com/ivanhofer/typesafe-i18n/tree/main/packages/runtime#plural)

And also a script that takes an CSV file for a translation and automatically generates the local_translations.h and /18n/.../index.ts files

[tp1de]

Do you like to check the translations and make suggestions? You have to download and compile the v3.5.0 branch.

I am happy to support. Do I have to download v3.5.0 branch separately or can I include it in my forked repro? Please advise

[MichaelDvP]

Do I have to download v3.5.0 branch separately or can I include it in my forked repro?

Both is possible. I think best way is to fetch v3.5.0 branch from ems-esp32, make/correct translations in this branch. Then merge it to your forc. This allows to PR only the translations.

[bbqkees]

@proddy indeed the easiest would be a stub. Otherwise I may miss some stuff.

[MichaelDvP]

I've found some more missing translations. I'll add them and prepare the nl/index.ts with original english texts. For the entities in local_translations.h should i add empty nl-translations F(""), or F("NL"), so you can search if you missed one, or leave it as lists (tag, en, de) and you complete to (tag, en, de, nl)

[proddy]

Ok thanks. I would just leave it and me or Kees can add the extra value. It's also worth checking if NL is chosen and it can't find a translation it's doesn't crash. I think I added a check to the code but can't remember. I wrote that piece on a plane

[MichaelDvP]

Yes, you have a check that defaults to en if translation is missing. Naming multiple entities "NL" will give strange results when sorting, but it will not crash. I'll give only an example for the first list: MAKE_PSTR_LIST(on, F("on"), F("an"), F("naar")) (translation from deepl).

[proddy]

And enums too if a translation is missing?

[proddy]

Also if a translation is missing we can prefix the default EN string with a message

[MichaelDvP]

Yes, enums working with default fallback.

I'm struggling a bit with the enum/bool formats and translations.

• Web: is clear, bool as selected, enum translated.
• console show: entities are translated fullnames, i suggest bool and enum also fixed to translated text (bool as web format), now it is like mqtt-format.
• api/device/info: shows translated entity fullnames, use the api-settings (maybe index/number) or fix to readable translated?
• api/device/values: shows shortnames and should output the api setting format, translate enum/on/off or not (fixed to en)?
• mqtt: shortnames, mqtt-setting format, translate enum/on/off or not?
• api/device/value: gives the complete value info with long and shortname and enum options. Translate?

I'm using numbered enum/bools now for mqtt/api, than i don't need to changes scripts when trying a different language.

[proddy]

I'd say translate everything if you can.

[tp1de]

I would not touch the entity names for mqtt / api.

[MichaelDvP]

Ok, i've pushed an update with nl selectable. @bbqkees Just edit the i18n/nl/index.ts for web and local_translations.h for entities.

@tp1de

I would not touch the entity names for mqtt / api.

Don't panic, your ioBroker-adapter is working well. Example enum:

{
  "_id": "ems-esp.0.thermostat.wwcircmode",
  "type": "state",
  "common": {
    "id": "thermostat.wwcircmode",
    "name": "ems: dhw Zirkulationspumpen-Modus",
    "type": "mixed",
    "read": true,
    "write": true,
    "role": "level",
    "states": {
      "0": "aus",
      "1": "an",
      "2": "auto"
    }
  },
  "native": {
    "ems_enum": [
      "aus",
      "an",
      "auto"
    ],
    "ems_type": "enum",
    "visible": true,
    "ems_command": "wwcircmode",
    "ems_device": "thermostat",
    "ems_id": "",
    "ems_api": "V3"
  },
  "from": "system.adapter.ems-esp.0",
  "user": "system.user.admin",
  "ts": 1662024523980,
  "acl": {
    "object": 1636,
    "state": 1636,
    "owner": "system.user.admin",
    "ownerGroup": "system.group.administrator"
  }
}

[bbqkees]

I'll do the translation this weekend.

[tp1de]

@proddy @MichaelDvP Any chance that you explain a bit more in detail (for a non-IT guy like me) how to proceed to get v3.5 into my forked repo. Sorry .... I am lost for the moment

[MichaelDvP]

@tp1de I've made a PR to your dev-branch. But it needs some manual merge. The strings are moved to 'locale_common.h/local_translations.handregister_device_value` is changed, mainly remove the nullptr from bool-types. For testing you can also make a new branch and merge there (and close my pr).

[MichaelDvP]

In network settings and status there are some missing translations: IP Address, Gateway, Subnet, etc. Translate now, or, as they are very common, skip, or wait until we have more space in flash?

[proddy]

I originally left them out as they're quite universal I believe. But for completeness we can add them. There's plenty of flash space over I believe

[MichaelDvP]

There's plenty of flash space over I believe

You're right, the dutch adds around 15k and we have ~150k free, no problem to add more languages.

[proddy]

nice, that gives us plenty of room for the next set of languages (swedish, polish).

[proddy]

@MichaelDvP I made some changes to the signin page to show country flags. I also changed the way the text is shown on the Help page as @bbqkees had challenges using the same wording context. It does mean these HELP_INFORMATION_1 to 5 need re-translating into DE and NL again. sorry about that.

[MichaelDvP]

Good, the help info with fragments needed some iterations and checks to make it read like a german sentence. I think EN-flag should be GB, i doubt Bosch sells EMS-heatings in US.

[proddy]

Good, the help info with fragments needed some iterations and checks to make it read like a german sentence. I think EN-flag should be GB, i doubt Bosch sells EMS-heatings in US.

hehe, you're right. I'll change it to UK and use the GB flag. BTW I'm taking the SVGs from https://gitlab.com/catamphetamine/country-flag-icons/-/tree/master/3x2

for the record, I do recall Kees selling a few units to the US and Canada. They should be happy with the Fahrenheit code you added.

[proddy]

I've added stubs for Swedish (SE) so eltoro and henrik (from Discord) can contribute. The two files that need changing are interface/src/i18n/se/index.ts and src/local_translations.h