[Proposal] Enhanced Translator

[eouia]

MagicMirror² Enhanced Translator

This project is rewriting the internal translator class of MagicMirror to have enhanced features.

• dev repository : https://github.com/MMRIZE/MagicMirror_EnhancedTranslator
• dev branch : https://github.com/MMRIZE/MagicMirror_EnhancedTranslator/tree/enhanced-translator (original based 2.16-develop of MicMich/MagicMirror)

Motivation

For several years, the current translator class works for L10N/I18N of MM successfully. But there is still some lack of features. Especially in the below cases, we need more improvements.

• Maintenance of translation files.

  Whenever a new translation by the user is added or updated, translations.js(core) or .getTranslations()(module) should be updated and released officially to prevent the update-conflicts issue.

  By versioning up, some old translation might be obsoleted or needs to be updated for the changes of terms used in MM.

  And also, it is hard to add a light-modified custom dictionary like de-au.json. It would probably be copied from the original de.json, and even though the difference might be very little, but full de-au.json should be published and maintained.

• Locale-aware

  One xx.json cannot hold several locale-specific translations difference. US English and Australian English are noticeably different from each other from vocabulary to grammar. Some Australian users may prefer "G’day mate" instead of "Hello". Some British people might not be comfortable with the spelling of "Color".

  English users in Canada(en-CA) notate dates as 4 September 2021 or September 4, 2021, And French in Canada(fr-CA) notate date as 4 septembre 2021. and the usual numeric format is 2021-09-04. But in the US(en-US), people genrally use 09/04/2021. And in France(fr-FR) it is 04/09/21.

  German people prefer to write a number like 1.234.567,89, but in the US, it will be 1,234,567.89. In India, it will be 12,34,567.89. In China, under 'hanidec' numbering format, it will be 一,二三四,五六七.八九.

  Generally, MM module developers have tried to solve these issues by providing specific additional L10N converted values by config definition. Occasionally it makes another too-many-config-value issue.

  And for the date format, we are facing momentJS deprecation. (Well, luxon might be the excellent alternative, though.)

• For multi-lingual user (too-simple-fallback)

  Current fallback mechanism - primary language or 'en' is not enough. Some people use multi-languages in their country.

  Certain Serbian users can speak Serbian, Croatian, Romanian, Hungarian, Slovak, Czech, etc. Anyway English might not be his prefer language. With luck, some modules might provide hu.json. and some modules might provide hr.json. But he should select only one language for his Mirror. And whatever he chooses, other modules will display English. Of course, en must be the last-final-safe fallback. However, it will be convenient and user-friendly if MM provided more steps before the final solution.

  Also, there could be cases like this; sometimes fr-CA user needs fr-CA translation at first, then want to use familiar fr, then en-CA if available, at last reluctantly use standard en as a final fallback.

• flexibility for natural language process

  translator is not a template engine. However, if it could handle Object and Array it would provide more abundant natural translation results. In some countries, the first name is preferred to the Family name, and in some countries, vice versa. If translator can handle user: {firstName, familyName, gender, title} together instead of addressing each userFirstName, userFamilyName, userGender, userTitle, it could reduce the codes and make meaningful context.

  Customizable value-formatters would be a help for L10N/I18N also. A Module developer supports only general value and format, and then a translation provider could convert that value to another format with those formatters on dictionaries.

  For example; Instead of "Unread mail: 3", more natural sentences "There is a mail unread.", "There are 3 mails unread." or "Es gibt eine ungelesene Mail.", "Es sind 3 Mails ungelesen" could be possible with some formatters and language/locale information by translation provider, not by module developer himself.

  Finally, a translation could be used as a light template engine with those features. It will give more freedom to the user who wants to customize the MM.

Those are the main reason why this project begins.

Improvements

• Multi-language fallback by prefer order and related-locales
• Translation dictionaries autoload
• Handling Object and Array as replacement variables
• custom value formatting

.... moe details are here - https://github.com/MMRIZE/MagicMirror_EnhancedTranslator/blob/master/README.md

By the way, I haven't made this as a PR yet, because it will fail to pass the current test. For the test, MM's most parts would be running on the browser, why we need nodev12 test?

[khassel]

Link to forum discussion

[rejas]

Thats an impressive proposal :-) Having a more flexible translation gets a big plus from me!

Question: Would current translation files / module still work?

[MichMich]

Indeed. Super impressive. But as stated: it's important that it is a not breaking change.

[eouia]

For backward compatibility;

• Translation file format is not changed. formatter and Obejct & Array properties is added but nothing is dropped. The old translation file still is usable (even file naming rule also still be the same. - But as I wrote in the document, kr.json is the wrong name for Korean language translation. It should be ko.json)
• config.language is still live. The new Translator will use config.language as the primary language prior to the newly suggested config.languages: [].
• module.translate(key, [replacementVariablesObject], [FallbackMessageString]) works. (internal logic was changed, though.) And added one more parameter asObject, but it is optional.

Changed;

• Translator loading and initializing behaviours are changed. In main.js and module.js, related codes are modified. As result, the old .loadTranslations() was obsoleted. Anyway, that method was used in only main.js and module.js And to change behaviours is this proposal's main purpose.
• module.getTranslations() in current modules will be ignored. Dictionaries will be loaded automatically by main config.language & config.languages
• As similar, translations.js will be ignored.

With this patch, MM 2.16 with default core modules was successfully played. Not tested many 3rd party modules, but the All I test looks worked.

I rewrote the unit test and passed. but for the e2e test. I have no experience with that, so I failed to pass the e2e test. I have no idea how to adjust this e2e test suite for my patch.

[eouia]

@MichMich @rejas @khassel and all who have an interest in this proposal;

Even if this project might have the worth to be included in the mainstream of MM, It would be better to wait to MM 2.19(or 2.20?) - 2022 Apr., 6 months later.

• At that time, node v12 life ends. (new I18N/L10N featureIntl would be the new standard in the JS environments at that time. node 12 doesn't support Intl.)
• Not related directly, but to be considered; MM's default I18N/L10N dependency - MomentJS's destiny. Will MM drop the dependency of moments off? I think whatever decision will be done until that time. By the decision, some core modules might have to be rewritten. (And probably related config value/format should be adjusted by that decision.)

So I'll close this issue at this moment, but I'll keep the progress in my own repository to wait in the meantime until next April. If you have any suggestions or discussions, leave them in my repository. I'll reopen this issue at the proper time.

https://github.com/MMRIZE/MagicMirror_EnhancedTranslator

Thanks.

[MichMich]

@eouia, I really appriciate your work on this. Indeed it might be a good idea to wait until Node's v12 EOL. My proposal is to merge it into develop directly after we released a new master that way we have 3 months to test the new setup before we need to release a new master.

I have no problem dropping the use of Moment.js in the MagicMirror² core if we have a good alternative. But we might want to keep moment.js part of the default dependencies for the 3rd party modules.

It might be a good idea to open an issue in this repo with info about your work, so there is one central place we can discuss planning, issues and features. But I leave that up to you if you want to.

Cheers! And thanks again for your awesome work.