Translation system proposal

[clsource]

I believe that Masonite would benefit having a good translation system.

Several ways of doing it exist today. Like getttext (mo and po files) and other methods like using code files and variables.

I have used Processwire CMS for quite some time and it has the best translation system that I have encountered.

It consist of similar functions to gettext but with json files instead of po and mo.

You can read the extensive documentation here

https://processwire.com/api/multi-language-support/code-i18n/

The implementation here

https://github.com/processwire/processwire/blob/master/wire/core/LanguageFunctions.php

A small improvement that I wrote about here

https://medium.com/@clsource/better-translatable-strings-in-processwire-621e9e6b18ee

And example translation files here

https://github.com/processwire/processwire/tree/master/site-languages/install/files/1012

Basically the idea is having a new folder named i18n for storing the translation files. Then inside that folder will be another named after each desired language code. the name 'default' will be for storing the default strings.

When using the translation functions will look for the desired translation. This could depend on a middleware that changes the translation depending on route params or defined explicit by the developer. If no translation is found then default is used. If no translation is found in the default then the function should return the input string.

For creating the translations a craft command will scan all project files and generate the json files in the corresponding folder.

Example $ craft i18n es

Will scan the files and create Spanish (es) folder with the jsons for each textdomain found.

If you want to use Spanish as the default you could overwrite the files in the default folder or force the config in a Masonite config var.

maybe using something line $ craft i18n default es

Processwire has a good GUI for translations but since Masonite is more a framework than a CMS

Maybe using a human format like https://hjson.org/ could be more user friendly than raw json files.

Cheers :)

[clsource]

I made some investigation of different systems

https://laravel.com/docs/5.7/localization https://guides.rubyonrails.org/i18n.html http://airbnb.io/polyglot.js/ https://github.com/tuvistavie/python-i18n https://docs.djangoproject.com/es/2.1/topics/i18n/ https://www.i18next.com/overview/introduction https://flowframework.readthedocs.io/en/stable/TheDefinitiveGuide/PartIII/Internationalization.html http://cldr.unicode.org/ https://github.com/twitter/twitter-cldr-rb http://babel.pocoo.org/en/latest/

I believe Masonite should provide a translation system simple to use and with the less amount of configuration possible, convention over configuration. With that in mind, it will strive to be the mix of the best ideas of different i18n implementations.

The basic idea is give users an opinionated translation framework with the common needs covered. For more complex situations other fraemworks like Babel could be used.

The translation system would only consider storing raw texts and not formatting.

Translations files

Similar to Laravel, the translation files will be stored in resources/lang.

Following Processwire's logic there should be a default folder, if this folder is present then the translation files inside will be considered as the default ones. Other translations will be stored in other directories. This could also be overrided in a configuration but it would be optional to do so.

Translation file extension will be .py as they will be a simple python code containing a dictionary. Like Laravel's php files.

There must be a __init__.py file containing basic language data to be detected as a valid translation.

Naming convention

All files would be stored in the same folder and the naming convention would be the following:

• Directory separators (/ or \) would be standarized as double dash -- except the root that would be ommited.
• Extension dot (.) would be replaced with a single dash -.
• Trimmed spaces.
• Lowercase.

Example:

./resources/templates/welcome.html would be standarized as resources--templates--welcome-html.py

init.py

The file content of __init__.py must contain two properties and one optional:

name = 'en'
title = 'English'
enabled = True # Optional

Other properties (currency symbol, date format, money format, etc) could be added but that will depend on each user needs.

name will be used for identification in routes and other operations. Must be url encoded.

title would be used for general description in logs or other places.

enabled would be used for determining if the language should be available. If not present it would be considered enabled = True.

Translation file

The content of each translation file would be similar to the following example:

resources--templates--welcome-html.py

item = 'resources/templates/welcome.html' # Required
textdomain = 'resources--templates--welcome-html' # Required
language = 'English' # Optional
translations = {
    '<sha256>' : {
        'original' : '', # Optional
        'comment' : '', # Optional
        'note' : '', # Optional
        'text' : '' # Required
    }
}

item

Would store the path to the file beign translated.

language

The language of the translations. Used for informational purposes.

textdomain

Inspired by Processwire. Textdomains are used to ensure that only the necessary translations are kept in memory at the same time, and that there aren't namespace collisions of unrelated translations.

Each file is considered it's own textdomain and the textdomain is nothing more than the filename (including path) from the root of the Masonite installation. The textdomain is not loaded by Masonite until a function call from a given file requests a translation for a phrase. The textdomain consists of all translation phrases for the current language in one Python file.

The developer using translation function calls does not have to think about textdomains, as it is something that Masonite figures out behind the scenes. However, if a developer does want to override the textdomain from the curent file for a given translation, they can do so by specifying the filename (including path) in the function calls.

translations

Would store the translations for the textdomain inside a dictionary. Each dictionary key would be a sha256 hash of the original text.

• original : Stores the original text for translation.
• comment : A comment to help translation process.
• note : An additional note for complementing the comment.
• text : The translation for the original text.

I would later continue with the functions and craft commands 👍

[josephmancuso]

This sounds awesome although I don't think it should be in the core repo. This sounds like it's gonna be more than a few files.

I say we make this a separate package and then add it as a dependency for all new projects for 2.1. This way it is "in core" but it will live in it's own repository and be separate. Then we can just point a series of versions like we do with a lot of other packages.

If you want to start up a repo for this just drop a link to it in Slack and then we can all start working on it. I think a lot of this can actually be simplified down which is even better.

This is really good planning and we can use this as a great foundation. Good job checking all the other frameworks and libraries out there.

[josephmancuso]

Also I LOVE starting at the end result and working backwards so how do you want this to look inside the:

• view
• controller
• config file

code examples are awesome if you can supply them. Will it look something like:

<h2> {{ locale('en.welcome') }} </h2>

?

[josephmancuso]

also the resources folder will look something like:

app/
config/
resources/
  en/
    __init__.py
  ch/
    __init__.py
  es/
    __init__.py

?

[josephmancuso]

What i usually do when building a package is actually just make a simple module and put all the code in the module, tests etc. and then simply take the module and put it in a new directory and build a package out of it so if you want to just make a new application and we can implement the package based off of that, work out any kinks etc. unless you have other ideas for creating the package

[josephmancuso]

I can set that all up if you want.

[clsource]

Yeah a separated package could be used 👍 .

Here are some ideas for the structure and functionarlity

The base structure for translations will be similar to this

app/
config/
resources/
    lang/
        default/
            __init__.py
        ch/
            __init__.py
            resources--templates--welcome-html.py
        es/
            __init__.py
            resources--templates--welcome-html.py

In this case english is the default (No translations). Using this structure any language could be the default translation, just put the correct files inside the default folder, no need for configuration. Optionally it can be configured to another default directory if desired too.

Inside the config/application.py

    '''
    |--------------------------------------------------------------------------
    | Application Locale Configuration
    |--------------------------------------------------------------------------
    |
    | The application locale determines the default locale that will be used
    | by the translation service provider. You are free to set this value
    | to any of the locales which will be supported by the application.
    |
    '''

    LOCALE = 'default'

    '''
    |--------------------------------------------------------------------------
    | Application Fallback Locale
    |--------------------------------------------------------------------------
    |
    | The fallback locale determines the locale to use when the current one
    | is not available. You may change the value to correspond to any of
    | the language folders that are provided through your application.
    |
    '''

    LOCALE_FALLBACK = 'default'

Basically any file could be translated. Since the translation files would be generated with a craft command that would detect translation function calls and create the respective translation files.

For example app/http/controllers/WelcomeController.py would be stored like

app/
config/
resources/
    lang/
        default/
            __init__.py
        ch/
            __init__.py
            resources--templates--welcome-html.py
            app--http--controllers--welcomecontroller-py.py
        es/
            __init__.py
            resources--templates--welcome-html.py
            app--http--controllers--welcomecontroller-py.py

Language Detection

Masonite must detect and set the locale automatically using three ways.

• The first is analizing the request params in the defined route in a GET param named locale.
• The second is analyzing the request header Accept-Language https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language
• The third way is determining the locale based on the IP Address and geo referencing the country. Although this method is the least reliable.
• If all these fails then the locale will be the default specified in the config.

The translation functions will use the detected locale as the base for obtaining the translation strings.

Translation Functions

The translation functions could be used in any part of the app like controllers, views or other parts. They will retrieve the correct translation for the desired locale and file.

__(text, textdomain = None, comment = None, note = None, locale = None)

Performs the translation. Returns translated text or original text if translation not available.

• text Required : Original text to translate
• textdomain Optional : Textdomain for the text, may be class name, filename, or something made up by the user. If omitted, a debug backtrace will attempt to determine it automatically following the conventions.
• comment Optional: Used for giving context to the original text
• note Optional: Used for additional context to the translators
• locale Optional: Forces fetching the translation for a specified locale, if not found uses the default locale.

Example Usage:

__('Hello this text should be translated')

Inside views

{{ __('Hello i18n') }}

Forcing a locale that is not the default or autodetected

{{ __('This should be in Spanish', locale = 'es') }}

Using with string params

{{ __('Hello {name}').format(name='Hodor') }}

_i(intervals, count, textdomain = None, comment = None, note = None, locale = None)

Performs a translation depending on intervals. Uses the __() function for the base translation.

• intervals Required: A dictionary with intervals as the keys.
• count Required: How many items.

Example Valid Keys

0 : No items,

1 : Exactly the number of items defined. In this case just 1

1,19 : Interval from 1 to 19 inclusive

20,* : Interval from 20 or more items

Example:

{{ _i({'0' : 'There are no items', '1,19' : 'There are few items', '20,*' : 'There are many'}, count=2) }}
# Returns translated 'There are few items'

_n(intervals, count, textdomain = None, comment = None, note = None, locale = None)

Performs a language translation using a pre defined interval for keys Uses the _i() function for the base translation.

• intervals Required: A dictionary with intervals as the keys.
• count Required: How many items.

Valid keys

0 or zero : No Items Interval 0 1 or one : One Item Interval 1 2 or two : Two items Interval 2 + or few : Interval Default [1,25] Items. This interval could be configured in the __init__.py of the language * or many : Interval Default [25,*] Items. This interval could be configured in the __init__.py of the language ? or other : Other Interval. This interval could be configured in the __init__.py of the language

Example:

{{ _n({'0' : 'There are no items', 'few' : 'There are few items', '*' : 'There are many'}, count=2) }}
# Returns translated 'There are few items'

_p(text_singular, text_plural, count, text_empty = None, textdomain = None, comment = None, note = None, locale = None)

Perform a language translation with singular and plural versions. Uses the _n() function for the base translation.

• text_singular Required: Original text to translate in singular form
• text_plural Required: Original text to translate in plural form
• count Required: How many items. Used for determining if is plural or singular.
• text_empty Optional: If not set it will be used the text_plural as the default message for no items found.

(other params are the same as __())

Example

quantity = 4
_p('There is one item', 'There are {} items', count=quantity).format(quantity)
# Returns There are 4 items

[josephmancuso]

Seems good to me and it seems like you know a lot about this subject, way more than I do so just let me know where you need me and I'll be more than willing to help you complete this package.

[clsource]

I created a repo for testing this out

https://github.com/clsource/masonite-i18n

I'm still learning so any comment would be greatly appreciated 👍

[theodesp]

Jinja already has i18n support with: http://jinja.pocoo.org/docs/2.10/extensions/

and through the usage of trans tags:

http://jinja.pocoo.org/docs/2.10/templates/#i18n

There is no need to reinvent the wheel here and I think a simple approach will work fine. We need to hook some craft commands to handle i18n via babel.

[clsource]

Yeah babel is an standard way to bring i18n and localizations to a Python project. But I believe i18n can be easier to accomplish. Specially since most projects only needs basic text translation. The good thing here is that Masonite could benefit from both approaches. This i18n implementation will not be in the core and people could choose to use this simple version or a more complex one using babel. If you have the time, please bring babel to Masonite too :)

[josephmancuso]

since it's being worked on we can close this issue. We can still converse here though but it will be on a closed issue