Kibana Globalization

[srl295]

Kibana Globalization

Note: These requirements were originally written up over a year ago. Take any specific implementation details with a grain of salt. Phases that were already implemented should be reevaluated to make sure they still make sense, especially in light of the new kibana platform work.

Purpose

• Provide translation readiness in Kibana for view text (user data is a separate issue)
• A core translation layer that is framework agnostic and consumable in all parts of Kibana
• Provide plug-and-play capability (i.e. code free) for localization engineers when providing translations of different languages
• Delivered in a phased approach

Architecture Diagram

This was our original plan, but it might need to change dramatically for the new platform

Design

Phase 1

Note this phase was completed, but it's possible the implementation might need to change for the new platform. Most of the code lives here

Purpose

• Implement the internationalization engine (I18n class) which provides a means to supply translations in a standard format that it not dependent on a localization framework
• i18n engine manages all locale translations; registering all translations and loading the correct locale when required
• Translate the Kibana welcome message which proves that the I18n class provides translations as registered for Kibana

Decide Language for Translation Algorithm

• The algorithm which decides the language for loading translations is as follows:
  1. First, do a direct comparison with the highest priority locale in the HTTP header “accept-language” against the registered translation languages. If comparison found then this is the language returned.
  2. Next, do a partial comparison of this locale whereby you try and get a language code comparison against the registered translation languages. For example, "fr" or "fr-FR" is used for all regions of French if comparison applies. If comparison found then this is the language returned.
  3. Next, repeat step 2 and if need be step 3 for the next highest priority locale in the “accept-language” list.
  4. Continue, until match found in step 2/step 3 or end of the “accept-language” list. If no match found then return empty language.

I18n Class

• Manages the language translations for Kibana

• Responsible for loading translated content per language

• The translations file are JSON files with a flat keyspace, where the keys are unique. This uniqueness between translation plugins could be achieved by prefixing the keys with the plugin name. The key signifies the translation ID which would be referenced in translatable files (like JS, HTML etc.).

• The key value is the translation string

• Example translation JSON file

en.json

    {
        "UI-WELCOME_MESSAGE": "Loading Kibana",
        "UI-WELCOME_ERROR": "Kibana did not load properly. Check the server output for more information."
    }

• Core Kibana plugins like ‘kibana’ and ‘status_page’ could come with their own English translations bundled in
• API:
  • Return all translations for registered locales
    • getAllTranslations()
    • A Promise object where keys are the locale and values are Objects of translation keys and translations
  • Return all translations registered for the default locale:
    • getTranslationsForDefaultLocale()
    • A Promise for an object where keys are translation keys and values are translations
  • Return translations for a suitable locale from a user side locale list of BCP 47 language tags:
    • getTranslations(...languageTags)
    • A Promise for an object where keys are translation keys and values are translations
    • This object will contain all registered translations for the highest priority locale which is registered with the i18n module
    • This object can be empty if no locale in the language tags can be matched against the registered locales
  • Register translations:
    • registerTranslations(\<absolute_path_to_translation_file>)
    • The path to the translation file is registered with i18n class

UiI18n Class

• Handles the interaction between the UI/server and localization (I18n class)
• Maps the accept-language header to BCP 47 tags
• Fetches the language as requested where the locale is decided by the translation algorithm
• It also substitutes any missing translations with the default locale translation

Tool for Verifying All Translations have Translatable Strings

• Grunt run task tool that tests all translatable strings have a translation (i.e. all translation IDs have a corresponding translation string)
• This could be run by CI to verify globalization end-to-end capability
• A possible solution:
  • For non-angular constructs: Enforce a pattern to be used. For example a i18n(<key>) function in the Jade template. A tool can then be used to find such pattern and extract the keys to file
  • The keys in the key file(s) would then be checked against the language translation files registered

Deliverable

• Translate the start-up message (“Kibana is loading ...”) and error message in the Jade template (Kibana Jade template)
• Translation message will be loaded using the language as per the deciding algorithm. For phase 1 the default 'en' langauge will be used.
• I18n unit tests
• Tool can be run by CI to verify translation IDs have a corresponding translation string
• Kibana core plugin registers its translation file during the initialization phase. The translation file contains strings for the welcome message and the start-up error message.

Phase 2

Purpose

• Integrate a framework like i18next or formatJS with the i18n engine, or build our own. i18n framework needs to be independent from any UI framework (angular, react, etc).
• Provide example for enabling translations in an AngularJS view and a React view by creating a POC for each
• Implement a tool to generate the translation plugin
• Documentation

Translations Available Client Side

• Make the translations available on the client side by:
  • Embedding the translations in the initial HTML payload:
    • Reduces complexity and removes a round trip
    • Evaluation required to test if it performs well and does not create a large payload
  • If the evaluation of HTML payload is non performant, then the following approaches should be investigated in this order:
    1. Use current bundle mechanism:
       • (Kibana Jade template) to call API and generate the bundle which will be loaded during start-up
       • The JavaScript bundle produced will be of the following format: i18n_\<language>.bundle.js
       • Kibana loads all resource bundles on the client side after starting the single-page application
    2. Client side directly calls REST API and loads the JSON payload:
       • GET /i18n/translations ==> returns the English (or German or whatever) translations negotiated with the browser HTTP header “accept-language” priority list compared against the languages supported
       • GET /i18n/translationsForDefaultLocale ==> returns the default locale translations
  • All approaches will decide the language to be served up by using the language deciding algorithm (from Phase 1)

Translation Identifiers Added to Kibana UI

• Ids are added to the relevant view with Angular and HTML content
• English (en) translation file is generated for the Ids defined

BEFORE (HTML)

    <div class="sidebar-list">
    …

    <div class="sidebar-list-header">
        <h5>Selected Fields</h5>
    </div>

AFTER (HTML)

    <div class="sidebar-list">
    …

    <div class="sidebar-list-header">
        <h5 translate="FIELDS_SELECTEDFIELDS"></h5>
    </div>

Tool for Verifying All Translations have Translatable Strings (Update)

• Verify translation IDs in Angular templates
• The keys in the key file(s) would then be checked against the language translation files registered

Translation Plugin Generator

• Tool which generates a translation plugin
• Localization engineer should only need to add translation file(s) within the plugin directory, refernce the file(s) in the 'uiExports' plugin extension, and add plugin to Kibana
• All integration with Kibana works OOTB

Deliverable

• Angular and React views where globalization is enabled showing the pattern to use
• Verification tool updated to check translation keys in UI
• A generator that localization engineers can use to produce language translations and integrate them in Kibana in an easy manner
• Documentation relevant for the following stakeholders:
  • those that want to enable translation in Kibana
  • those that want to contribute translations to Kibana
  • those that want to create a Kibana Plugin

Phase 3

Purpose

• Add the translation identifiers and English translation strings for the existing Kibana views

Translation Identifiers Added to Kibana UI

• Ids are added to all the relevant UI content (HTML, JS etc.)
• English (en) translation file(s) are generated for the Ids defined

Deliverables

• English translation bundles are available so that localization engineers can start generating translation plugins for different languages

Phase 4

Purpose

• Contributing language packs

Handling Language Packs

• Provide a pseudo language which will test the potential real estate of views for all language
• Define core supported languages
• Process for language pack builders:
  • Creating packs
  • Merging packs

Deliverables

• Process for contributing language packs
• Core language packs as supported by Kibana

Blockers

Open Questions/Issues

• Translation of user data. TODO: requires a separate issue.
• Server side set locale (ignore the client's requested locale)
• Provide mechanism in UI for user to manually switch their language from list of supported languages
• What about bi-directional language support? This affects language, charts and UI content support.
• UI views should be able to handle different languages (This means when switching from one language to another (for example, English to German) that the look and feel is maintained) :
  • Different languages can have variable content lengths. This means having adequate spacing to handle the strings in each language.
  • Need to consider different fonts. Some web fonts won't support all possible languages.

History

This section has links to prior versions of this issue text.

• 2018 March 7th
• 2016 December 20
• 2016 December 14
• 2016 August 3
• 2016 August 2
• 2016 July 26
• 2016 July 20
• 2016 March 11

Authors

• Matt Bargar
• Martin Hickey: @hickeyma
• Scott Russell: @DTownSMR
• Shikha Srivastava: @shikhasriva
• Steven R. Loomis: @srl295

[Bargs]

Looking pretty good. Issue description updated, as well as my checklist. Just a couple of open items left.

[hickeyma]

@Bargs thanks!

[hickeyma]

@Bargs WRT "registering the core plugin translations" in delivery section of phase 1 of #6515, this could be updated as follows:

• Welcome message and start-up error message are registered with the i18n plugin. This will require separate mechanisms for the build and development process:
  • Build process:
  • Grunt task which calls the registerTranslations API and make them available to the build package
  • Dev process:
  • Registration of translations during dev start-up of the Kibana server (npm start). Hook is added to the 'optimize' module to call the registerTranslations API.

Additionally, maybe the "Blockers" section can now be cleaned up?

[hickeyma]

@Bargs WRT Jade template verification in delivery section of phase 1 of #6515, this could be updated as follows:

• Jade template verification of translation strings:
  • Enforce a pattern to be used. For example a translate(<key>) function in the Jade template. A tool can then be used to find such pattern and extract the keys to file
  • The keys in the key file(s) would then be checked against the language translation files registered

[Bargs]

@hickeyma lgtm - description updated.

[Bargs]

@hickeyma so I was thinking more about the core plugin registration problem and I'm starting to question whether we even need to build this intermediate json bundle per language. Due to the simple, flat translation file structure we've chosen we could easily build a single unified JSON object from multiple file streams on the fly. The i18n plugin would only need to keep track of the translation file paths per language. This would simplify a number of things:

• No need for a plugin install time hook. Translation file paths would simply be registered at server start up time.
• Core plugins and third party plugins would be treated equally, registering themselves inside the existing plugin init hook.
• If (if!) we run into performance issues, we can always just add a layer of caching.
• We get live reload in dev mode for free, because there is no build step.
• This is actually more in line with the direction we're trying to head with css/js resources as well. Ideally we'd like to remove "compilation" from production environments entirely.

What do you think?

[hickeyma]

@Bargs good feedback, thanks.

Are you saying that we now just register the translation files names per plugin when the plugin is being initialized? Does this mean the plugins still have to register with the i18n plugin but not during the install phase of the plugin? That would definitely simplify the infrastructure hooks difficulty. It probably doesn't matter then if you register just the filename or the actual file?

[srl295]

(I'm coming off of NodeSummit, just getting to some more detailed review.) @Bargs this makes some sense.

[Bargs]

@hickeyma yeah, basically the i18n plugin just needs a reference to each translation file and knowledge of which language it contains. I think each plugin would still be responsible for explicitly registering its translation files with the i18n plugin during init time, just like how plugins currently register their routes with Hapi at init time.

[shikhasriva]

@Matt and @hickeyma , this will simplify the approach for sure. And removes the install dependency. I do like this approach

[LeeDr]

I'm not clear if the initial phases are providing a translation at the server level and all users see only that one language, and then in Phase 3 it uses the browser HTTP header “accept-language” to translate per users browser language? Can we add notes to the phase descriptions to make it clear what the functionality is?

We might also want to start a discussion about which phases might be release candidates. It can happen later, but I wouldn't want a bunch of contributors to have an expectation about getting a release with some phase functionality and Elastic deciding not to release until some other phase. For example, translations for the whole server and not on a per-user basis.

[srl295]

a translation at the server level and all users see only that one language

I don't think this "one language" step is part of the plan. The current PR (phase 1) already translates, using accept-language, on a per-browser basis— it's just that only the "kibana is starting up" and one other error message are translated. Phase 4 has the "Translation Identifiers Added to Kibana UI" step which actually allows the rest of the UI to be translated. Hope this helps!

[hickeyma]

@LeeDr thanks for the feedback.

a translation at the server level and all users see only that one language

To elaborate on @srl295 description above, the idea here is to deliver in a phased approach. The phases 1-3 is to put the i18n infrastructure in place and show it working end-to-end in Kibana. The key part is the i18n plugin which provides means to supply translations in a standard format that it not dependent on a localization framework. The proof of this is to translate the welcome string in Kibana. Phase 4 is about translating the rest of the UI using the underlying i18n plugin to get the strings.

Can we add notes to the phase descriptions to make it clear what the functionality is?

Sure, can do. That is a good idea.

We might also want to start a discussion about which phases might be release candidates.

This is on the TODO list with @Bargs and @epixa.

[hickeyma]

@Bargs, @LeeDr I have updated the design doc kibana-proposal-update as per your feedback. Let me know if I have covered all the angles.

@Bargs, @srl295 could you update the main ticket description?

[srl295]

@hickeyma done!

[hickeyma]

Thanks @srl295

[hickeyma]

@Bargs, @srl295, could you also add this line to i18n Plugin (update) in Phase 2 of #6515:

• The algorithm which decides the translation language to use for retrieving the translations works as follows:
  • First, do a direct comparison with the HTTP header “accept-language” priority list against the registered translation languages.
  • Next, do a partial comparison whereby you try and get a language code comparison. For example, "fr" or "fr-FR" is used for all regions of French if comparison applies.
  • Finally, the default language as configured per Kibana instance is used. Default configuration is "en". This is also used if no HTTP header “accept-language” is set.

[hickeyma]

@Bargs, @epixa, @shikhasriva, @spalger, @ycombinator, @cjcenizal, @thomasneirynck, @tylersmalley, @bevacqua, @simianhacker, @BigFunger, @jbudz, @ppisljar Thank you all for attending the red-team review yesterday and for your feedback. Let me know if I forgotten anyone.

Here are notes from the review (thanks to @Bargs for taking notes!):

• [ ] Handling requested language for translations:
  • [ ] Server side set language (ignore the client's requested language)
  • [ ] Ability to define a set of supported languages on server instead of relying on list of languages auto generated from available translations (when some might be partial)
  • [x] In the "partial match" stage of the accepted language matching algorithm (Phase 2 'Decide Language for Translation Algorithm'). What would happen if a user requested 'fr-FR', but the system only supports other regional dialects. Should we have capability to set a default region for all language codes (e.g. fr, de etc.) requests?
  • [ ] Provide mechanism in UI for user to manually switch their language from list of supported languages
  • [x] What do we do when the requested language exists, but it doesn't cover all translations keys (for instance, there are French translations for a plugin but not core Kibana)? Do we fill in the missing keys with the default language?
• [ ] What about bi-directional language support? This affects language, charts and UI content support.
• [ ] UI views should be able to handle different languages (This means when switching from one language to another (for example, English to German) that the look and feel is maintained) :
  • [ ] Different languages can have variable content lengths. This means having adequate spacing to handle the strings in each language.
  • [ ] Need to consider different fonts. Some web fonts won't support all possible languages.
• [x] Use YAML instead of JSON for the translations? Easier to write strings than in JSON, less escape chars, multi-line, etc.
• [ ] Should we embed the translations in the initial HTML payload? Reduces complexity and removes a round trip. May eliminate the need for a HTTP API? Probably worth a PoC to see if it is performs and does not create a big payload?
• [ ] Need to consider how to translate and verify xplugins (React) (separate meeting w/ xplugin devs) Shouldn't treat x-plugins as a second class citizen after core Kibana.

Feel free to comment and give feedback especially if anything has been forgotten!

[hickeyma]

I have updated the design doc kibana-proposal-update with the feedback leaving unresolved items as open or blockers.

Maybe for the moment, we use it as an intermediate document and only update the main document when we are ready?

[shikhasriva]

Thats works fine. @hickeyma , Thanks for aggregating the comments

[srl295]

responding to https://github.com/elastic/kibana/issues/6515#issuecomment-237287195 very late

yaml instead of JSON…

I'd recommend JSON as it is more commonly used in translation systems.

[Bargs]

@ycombinator you had some compelling reasons about using YAML, could you share them here?

[ycombinator]

Sure, I don't feel strongly about either choice but, in a past i18n project YAML was chosen over JSON for a couple of reasons:

• Its a tiny bit more human-readable than JSON. This might lower the barrier to entry a bit for folks who'd like to supply translations but may not be familiar with JSON.
• YAML supports a few heredoc formats which make it more natural for expressing multi-line translations, as opposed to JSON where the newlines would have to be escaped. That being said, I imagine most translations for Kibana will be quite short so this is probably not a huge factor.

Those are the two reasons I recall at the moment. If I think of others I'll add them to the comment. But again, I don't personally feel strongly about one being strictly better than the other.

Also, we could go with JSON initially and start accepting YAML at a later point in time if we start to run into the above reasons in practice.

[hickeyma]

@ycombinator Thanks for the feedback. For now, we will use JSON. We can always provide a YAML to JSON converter if that is the preference for localization engineers.

[hickeyma]

What do we do when the requested language exists, but it doesn't cover all translations keys (for instance, there are French translations for a plugin but not core Kibana)? Do we fill in the missing keys with the default language?

Refer to i18n plugin pr comment for more details.

[hickeyma]

Decide the phases to be delivered in the Elastic release candidates

Discussed with @Bargs. Not necessary for the feature. More for Elastic internal QA.

[hickeyma]

@srl295, @Bargs I have updated the design doc kibana-proposal-update. Could you update the design doc when you get a chance?

[Bargs]

Updated

[hickeyma]

@Bargs thanks

[shikhasriva]

thanks ... great collaboration and awesome progress.

[hickeyma]

@srl295, @Bargs, @spalger I have updated the design doc kibana-proposal-update following some changes in Phase 1. Could you update the design doc when you get a chance pls.?

[hickeyma]

@Bargs, @spalger I wonder should we re-examine the design again following the merge of "Phase 1"? I started a Phase 2 PR (https://github.com/elastic/kibana/pull/8766) and @posijon is working on it. Maybe we should see if current design still applies for the phase?

[spalger]

It might be worth taking a look. There are things, like pulling the translations from a rest api, that I would push back on.

[Bargs]

Yeah we should definitely take a look and make sure we're on the same page. The REST API bit, for instance, I don't think has been updated since our "red team" meeting. At that time I believe we said we'd try embedding the translations in the initial page load first.

Also, issue description updated.

[hickeyma]

Great stuff @Bargs and @spalger. Lets try and sync-up on IRC and work out how to progress for phase 2 and 3.

[hickeyma]

@Bargs, @epixa, @shikhasriva, @spalger, @posijon Thank you all for attending the sync-up yesterday and for your feedback.

Here are notes from the meeting (let me know if I have misrepresented anything):

• It is good that we landed Phase 1. We want to now drive on with Phase 2 and 3.
• Phase 2 is to get the angular template right for localization engineers to use.
• Phase 3 is to add translation bundles for various languages.
• Updates to Phase 2:
  • In "Deliverable" section, "making the translations to client side" needs to be updated. The proposal is to now implement it by embedding the translations in the initial HTML payload. If discovered that this implementation is non performant then we will consider other approaches like: using current bundle mechanism or then using REST APIs to return the translations.
  • Add to "Deliverable" section, Elastic needs to handle how to translate and verify xplugins with React
• "Tool for verification of translations" needs to be moved from Phase 3 to Phase 2.
• "Implement a tool to generate the translation plugin" is to be moved from Phase 3 to Phase 4.

[hickeyma]

@Bargs, @spalger Updated the design doc after yesterday's meeting as kibana-proposal-update. Could you update the design doc when you get a chance pls.?

[Bargs]

Updated

[yuwtennis]

+1

[hickeyma]

@spalger Updated the design doc kibana-proposal-update. Could you update the design doc when you get a chance pls.?

[spalger]

Could you update the design doc when you get a chance pls.?

Sorry, which design doc are you referring to?

[srl295]

@spalger he's asking for this issue ^^ ’s text to be updated from the above gist. As issue creator I was able to do it - so Done

[hickeyma]

Thanks @srl295

[hickeyma]

@spalger unfortunately no rights on the doc to update!

[srl295]

FYI: 🎦 posted blog+movie showing off g11n progress so far https://srl295.github.io/2017/03/17/translating-kibana/

[tangyong]

Great work so far! @srl295

[elkusr]

Thanks for all the great work! When can we expect phase 3 to be available?

[epixa]

@elkusr There's no target yet for phase 3, and there's still some work in phase 2 to complete first now that Kibana is using react so much.

[zzvara]

How are these efforts progressing now?

Is that correct, that the tasks presented in (https://srl295.github.io/2017/03/17/translating-kibana/) can be used to add any language to a Kibana instance (for example 5.6)? In other words, adding another language to Kibana is feasible as of this writing, it just takes a "little" effort, and that Phase 3 is about to ease this effort?

[epixa]

@zzvara It isn't possible to translate Kibana right now. Phases 2 and 3 need to be completed before you can translate templates across Kibana.