Technical design: static text localization and internationalization in 3.x

boutell commented 3 years ago

This issue concerns static text. For editable content localization see:

https://github.com/apostrophecms/apostrophe/issues/3205

The proposal we have arrived at internally is as follows, with additional details describing the process for shipping JSON files:

Use the i18next npm module server side, with namespaces. Specifically all Apostrophe phrases will be in the apostrophe namespace, regardless of origin. This keeps admin phrases separate from project level phrases.
Use the middleware support of i18next, and also inject a function to get the benefit of this in templates.
Push all of the apostrophe namespace phrases to the front end, via getBrowserOptions of the @apostrophecms/ui module, and configure vue-i18next with them.
Other apostrophe modules making up our UI will have to wait for i18next to initialize or they might not produce sensible output at startup. So we need to add a convention to control the order in which apostrophe’s frontend js apps (found in ui/apos/apps folders) are started so we don’t have to do a lot of listening on the bus. We will do this with a bit of new syntax. Rather than exporting a function, apps (like ApostropheI18nNext or ApostropheAdminBar) can export an object:

export default {
  priority: 1000,
  app() { ... }
}

Any app that exports just a function is considered to have a priority of 0. Then apps will be initialized in priority order, highest priority first. Initially it is expected that the only app needing to explicitly use this mechanism will be the app in the ui module that initializes vue-i18next.

i18n syntax examples

EXPRESS

We will use the i18next-http-middleware so we can write:

req.t('myKey', { ns: 'apostrophe' })

... Anytime we want to do this directly rather than in Nunjucks. If ns is not specified the default namespace is the project level one.

NUNJUCKS

We will inject a function that wraps req.t so that we can write:

{# apostrophe UI static text #}

{{ __t('myKey', { ns: 'apostrophe' }) {{

{# project level static text #}

{{ __t('myKey') }}

VUE

Via vue-i18next we will be able to write this in any Apostrophe admin UI Vue component:

{{ $t('myKey') }}

In browerspace the default namespace is apostrophe.

ui/src javascript

For now only ui/apos Vue components will have access to i18next phrases on the browser side. Support for access to i18next phrases in ui/src javascript may be added at a later time.

Other Modules, Other Namespaces

Additional npm modules providing add-on Apostrophe UI will need to be able to use their own namespaces, while still participating on the browser side. To address this, any module may set its own i18nNamespace option, like so:

options: {
  i18next: {
    ns: 'my-custom-namespace',
    browser: true
}

If browser: true is not passed then the phrases are only available server-side.

Special considerations for pieces

All piece types need their own labels, distinct from other piece types. So {{ __t('label') }} would not be a good solution in a Vue template as there can only be one i18next phrase with the key label. Instead, these labels will be localized server-side via req.t and made available via getBrowserData to the individual module as they already are today.

Keys versus default English text phrases

Apostrophe will ship with keys in i18next calls, rather than "default English" text phrases.

That is, a Vue component might contain: {{ $t('toggleSelectionOf', { label: this.label })

Shipping JSON files: Apostrophe core

The apostrophe core module and other apos UI npm modules will ship with JSON files in i18next JSON format, version 3.

These will ship in a locales subdirectory of the @apostrophecms/ui module, for the apostrophe namespace, and in other modules that choose to have i18next namespaces.

Within locales files we will follow the i18next version 3 format. For example, modules/@apostrophecms/ui/locales/en.json will contain:

{
  "toggleSelectionOf": "Toggle selection of {{ label }}"
}

Our intention is to add additional JSON files over time for other languages and locales. We welcome contributions to this project.

Shipping JSON files: other npm modules

Other npm modules that set the i18next option may also ship with a locales folder, which will automatically be loaded.

Shipping JSON files: project-level code

The project-level default i18next namespace, which is currently never pushed to browser-side JavaScript but is available for server-side localization in Nunjucks templates etc., can be populated by creating a locales folder at the root of the project.

Interactive Editing

It is our intention to eventually ship a module similar in functionality to the 2.x apostrophe-i18n-static module, which provided the ability to edit the translation of static text phrases as pieces in the Apostrophe UI. This will supplement the JSON support rather than replacing it altogether.

Distinct admin and project-level locales

In some cases, users with editing privileges may wish to view project-level static text in a selected locale without also translating apostrophe admin UI, even though translations are available. This is an enterprise requirement that has come up in the past. With a growing set of translations available it will essentially be necessary to support this feature, however this will arrive in a later phase of development. Initially, setting the locale to fr-fr will impact both the admin UI and the project level static text, assuming translations exist.

abea commented 3 years ago

This mostly sounds great. I think we can be clearer with naming the locales directory, though. Something like locale-strings, maybe. This is mostly a concern for the project-level since most devs won't ever see the core locale files. I'm not sure how strongly I feel about this. It is a win that devs won't all see a mystery locales directory appear in their project when they only have one locale.

boutell commented 3 years ago

We could call it localization.

On Wed, Jun 30, 2021 at 12:24 PM Alex Bea @.***> wrote:

This mostly sounds great. I think we can be clearer with naming the locales directory, though. Something like locale-strings, maybe. This is mostly a concern for the project-level since most devs won't ever see the core locale files. I'm not sure how strongly I feel about this. It is a win that devs won't all see a mystery locales directory appear in their project when they only have one locale.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/apostrophecms/apostrophe/issues/3204#issuecomment-871549273, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAH27LLGIVM2CNLYJQXSMDTVNAMDANCNFSM47QZKNBA .

--

THOMAS BOUTELL | CHIEF TECHNOLOGY OFFICER APOSTROPHECMS | apostrophecms.com | he/him/his

myovchev commented 3 years ago

l10n is a standard and sounds like a candidate for folder containing the... l10n :)

I was a contributor at Mozilla (translated the entire Mozilla and Firefox website in Bulgarian) few years ago. I can give you feedback about what challenges Mozilla had with l10n, how they were solving them in terms of tooling and what went wrong (while I was around).

And of course count me in for translation of the core in Bulgarian.

boutell commented 3 years ago

Thank you so much @myovchev!

I hear you (and Bea) regarding calling the folder containing the localizations i10n. I'll make that change.

The module is still @apostrophecms/i18n in keeping with the naming convention of i18next itself, and because it fits the convention we're talking about: i18n is the enabling of i10n, i10n is the actual translations.

boutell commented 3 years ago

(Note that any module will be able to have an i10n folder and contribute localizations.)

boutell commented 3 years ago

Also the per-module option will be called i10n, which will hopefully prevent confusion about whether the @apostrophecms/i18n module has some kind of exclusive right to have one (it doesn't, it enables all modules to have one).

boutell commented 3 years ago

For now I am putting a pin in the idea of support for declaring priority levels for the admin Vue apps for the sake of making sure $t is available when the first component wakes up, because it looks like the i18n module is initialized so early that it shouldn't matter. If it turns out to be necessary we can always introduce it later.

abea commented 3 years ago

@boutell Note that's l10n with an "L" for the localizations folder.

myovchev commented 3 years ago

I'm not an expert in any way, but I know (educated myself on the matter many years ago and it's not a fun reading) there is a clear difference between l10n (yes, lowercase L) and i18n. One may say l10n is part of i18n as the latter includes also date formats, number formatting, etc. So in general I don't think there is a conflict between the module name and the localization folder, in contrary - they sound completely in sync. But again, not an expert here :)

abea commented 3 years ago

My current understanding of the terms is that i18n would be the full system, including string translation, dates, etc as you mentioned. Then l10n is primarily the execution of translating and defining what is delivered as part of each locale.

So I think we're in agreement that a l10n folder works fine with an i18n module. And that nothing is clear in this area of tech design!

boutell commented 3 years ago

Did I use the wrong letter? LOL. Will fix.

On Mon, Jul 12, 2021 at 1:06 PM Alex Bea @.***> wrote:

My current understanding of the terms is that i18n would be the full system, including string translation, dates, etc as you mentioned. Then l10n is primarily the execution of translating and defining what is delivered as part of each locale.

So I think we're in agreement that a l10n folder works fine with an i18n module. And that nothing is clear in this area of tech design!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apostrophecms/apostrophe/issues/3204#issuecomment-878445134, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAH27N44AT5MJPXQVQKLXDTXMOLDANCNFSM47QZKNBA .

--

THOMAS BOUTELL | CHIEF TECHNOLOGY OFFICER APOSTROPHECMS | apostrophecms.com | he/him/his

myovchev commented 3 years ago

@boutell l10n - localization - l then 10 letters then n i18n - internationalization - i then 18 letters then n

And the sky is the limit a8e - apostrophe :)

boutell commented 3 years ago

a8e, matey!

On Tue, Jul 13, 2021 at 6:30 AM Miro Yovchev @.***> wrote:

@boutell https://github.com/boutell l10n - localization - l then 10 letters then n i18n - internationalization - i then 18 letters then n

And the sky is the limit a8e - apostrophe :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apostrophecms/apostrophe/issues/3204#issuecomment-878972722, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAH27KMBKZZ54IJNHMKAMTTXQIUXANCNFSM47QZKNBA .

--

THOMAS BOUTELL | CHIEF TECHNOLOGY OFFICER APOSTROPHECMS | apostrophecms.com | he/him/his

apostrophecms / apostrophe