sebmck commented 9 years ago

Consolidates #744, #39

morenoh149 commented 9 years ago

Keyword i18n i18next

tomasztunik commented 9 years ago

Happy to see it on the list for 0.4.0 - Would be amazing if there was a "tab switch" to change between different languages on create/edit forms if multiple languages were set for given model. Was going to look into that myself as I'm in dire need of that for my current project but currently can't really see an easy way to extend the fields without forking and mutating core code - bit afraid to break compatibility for updates at this early stage of 0.3.0 (still going over everything you've just released, great job with 0.3!)

What I'm looking into is elegant and "keystone" way for localised API responses for my SPA that would play nicely with the admin interface. It handles localised routes on client side but I could imagine that for server side rendered apps and pages also localised routes could be something desired.

Also localised fields would be awesome for non text types like money (item cost for shopping cart for example) or even localised images further down the line in case you wanted to prepare alternative images for different languages as sometimes cultural differences/fashion might make more sense for one language then for another.

Have you guys talked about any ETAs for multilang/i18n support? Do you have something in the works or planned already?

morenoh149 commented 9 years ago

I'm looking into this @tomasztunik would you agree that https://github.com/i18next/i18next-node and it's client-side version https://github.com/i18next/i18next are the most mature language libraries in the node ecosystem? https://github.com/krakenjs/makara looks good but not sure how tightly coupled it is to dust template language

tomasztunik commented 9 years ago

i18next looks good, used it quite some time ago in one project and it was solid back then. I'm pretty sure it only matured by that time. I know Makara only from reading about it as part of kraken but never used it. Looking at the docs it seems to be built more as an integral part of kraken rather than standalone component.

If you don't mind me asking - are you planning to work on translations for both static and dynamic content? And if for static as well how did you envision that?

Was thinking myself a bit about it and not 100% sure what would be the best way I'd approach it - now I'm thinking about adding i18n dict field type to allow to create and edit multiple dictionaries in case for example you were building SPA app and didn't want to load whole dict with the app but rather have it per module/component to be lazy loaded using custom api endpoints. This kind of special field could also maybe make it possible to register a dictionary under given namespace for use in the server side templates... but yeah - just thinking out loud :)

joaosamouco commented 9 years ago

Do you have an ETA of when keystone will fully support i18n? And how deep are you in the development of this feature?

We're looking into this but we do not wan't to dive into it if someone else is really close to achieve this and this feature could be released any time soon.

morenoh149 commented 9 years ago

hasn't been started apart from preliminary discussions. The details outlined here https://github.com/keystonejs/keystone/issues/744 are pretty spot on minus recent changes in the node ecosystem. If you'd like to start building this out please share a link to your feature branch if you'd like guidance.

Otherwise I could work on this in late April.

joaosamouco commented 9 years ago

We are willing to dive into this but we're still not sure how to handle both static and dynamic content. I've been reading all these issues related to multilanguage support and you mostly discuss lib related stuff.

Some people said something kinda related to how to implement this with keystone but nothing to specific and I'm still not sure if any of you came to any conclusion about how this should be designed and implemented inside keystone.

I want to be sure that we follow the right approach and this fits perfectly with keystone. Did any of you guys that have looked into this issue thought about how we will handle dynamic content and implement everything overall?

It's clear that static content inside the views will be tackled with the classic approach of wrapping that same content in a translation function (e.g. __). Though, dynamic content is way more tricky. Any thoughs?

@morenoh149 @sebmck @JedWatson

morenoh149 commented 9 years ago

@joaosamouco it was fleshed out pretty well by @sebmck There's probably a way to detect the language requested by the end-user http://devdocs.io/express/index#req.acceptsLanguages and/or enable an easy way to specify all routes under a path should be a specific language e.g. foo.com/en/... and foo.com/es/... The latter is trickier.

danielkhan commented 9 years ago

Hello. I hope it's ok if I jump in right here.

Locale on path and detection: Look at i18n-abide. They do it quite well. I'm using this on http://crowdlynx.com. The important part is to remove the language url part after registering the language and before passing it down the routing chain.

Translation of statics: Also i18n-abide - they use jsgettext for string replacement. It is easy to use this with translation tools like phraseapp then.

Content Translation: Why not add a i18n attribute to the schema declaration? If it is set there will be the field with subfields for every language. Like:

headline { de: 'Heute ist ein schöner Tag', en: 'Today is a good day' }

I would also recommend adding a switch to explicitly add a translation to an entity because often not every content is available for every language.

Just my 2 cents.

magalhas commented 9 years ago

@danielkhan the i18n field is a good idea, but what happens afterwards? In some use cases you want to translate in the backoffice, in other use cases you want to provide a translation file to third party service.

danielkhan commented 9 years ago

@magalhas - usually translation offices can deal with CMS systems but anyways - if we take it to this level we are talking about workflows for translation, approval, etc. I won't take it that far. Usually it's enough to use translation files for statics and and i18n-able database schema. This is how it works in other systems like SilverStripe or Typo3. They all offer some kind of mechanism to create language mutations and then offer a easy view that shows the master language along with some input element to insert the translation.

woodyrew commented 9 years ago

I've not had any experience with i18n but will most likely be doing a bilingual site soon. If it's in a state to test I'll happily help out with bug fixes, let me know a tag to checkout if so.

tinganho commented 9 years ago

Hmm, I'm maybe a little bit late in discussion. I'm the author of an i18n tool called l10ns. I'm also apart of the javascript-globalization group. I also happen to recently be a user of Keystone. I would be happy to help with any design around Keystone's i18n part.

I think it would be good if you choose a solution that's supports the standard ICU messageformat. It's a DSL for localization. And it support more complicated scenarios than i18next. ICU messageformat might also soon be a proposed as a native object to ECMAScript(there exist one strawman draft).

I think Keystone should probably separate localizing labels(label text, button text etc.) and localizing content(bread text, titles, header or everything that exists in the mongodb). I think Keystone's job is to localize content only and localizing labels should be done by an external tool. When we query an object we can pass in a locale to get different localized content.

I think it would be good if you take a look at https://github.com/SlexAxton/messageformat.js. Because it might suit your needs.

joaosamouco commented 9 years ago

@tinganho Nice contribution to this topic.
I took a look into your tool l10ns and it looks really good. My only issue is: how hard is it for a non-technical translator to work with the tool interface and do his job? Did you ever hired some translator that had to work with your tool? How was it?

tinganho commented 9 years ago

No it is quite easy. Though for more complex translations they might ask a few question about the syntax.

LorbusChris commented 9 years ago

http://formatjs.io/ by Yahoo

ericelliott commented 8 years ago

I'm looking at starting a project which requires i18n. I'm very interested in formatjs. Just need to be sure that our translation service will play nice with the message format. Thanks @LorbusChris!

ericelliott commented 8 years ago

BTW, anybody interested in helping with this, I've been investigating options today. Will probably have a locale negotiation PR landing soon. I'd love it if some of you would like to review PRs or contribute to the effort.

If you don't mind me asking - are you planning to work on translations for both static and dynamic content? And if for static as well how did you envision that?

I plan to use translations for both static and dynamic content, but I'll probably want @JedWatson to be involved in planning dynamic translation support.

I agree we may want separate mechanisms for them. Dynamic content will probably require fewer features re: pluralization, grammar rules, etc, and instead just point to alternate db resources (a different text blob, different image URL, etc...).

Whereas static content, such as displaying UI labels, relative times, pagination numbers, etc... will require more thorough support, such as ICU messaging, etc...

We may also need to load alternate CSS / image resources, etc... depending on locales to cover things like right to left text display formatting, different length labels, and images containing text.

ericelliott commented 8 years ago

Just hit a small roadblock. Smartling, the translation provider that we're looking at, does not support ICU. There may be a way forward though by compiling to/from a supported format via the message format parser...

ericelliott commented 8 years ago

i18n checklist

[ ] Locale negotiation - Express support doesn't handle fallbacks well. See locale
[ ] User preference override - Some users may want to select a different locale - we should provide an API mechanism that will record that user's preference and use that instead of a negotiated locale. We'll need to discuss how to handle the "bring your own User model" case - maybe the BYO case will need to (optionally) provide a locale update mechanism.
[ ] Polyfill Intl in Node (doesn't come with all the required locale data). (all supported locales)
[ ] Polyfill Intl in clients (only the user's locale)
[ ] Translations and message format for static UI messages (requires support for numbers, dates, names, currency, plurals, etc...). FormatJS looks like a great option, but only supports ICU message format, which is problematic for me because we're looking at Smartling, which lacks ICU support. See my comment above.
[ ] Translation handling for dynamic content - will likely integrate with the list API, offering alternate db records for alternate translations.
[ ] Note to self: read https://github.com/keystonejs/keystone/issues/744 and see if there's anything I need to pull into this list.

I need your help

I'm new to Keystone. Please feel free to suggest other "see also" links containing any information about existing support for i18n concerns in Keystone.

ericelliott commented 8 years ago

@danielkhan

Content Translation: headline { de: 'Heute ist ein schöner Tag', en: 'Today is a good day' }

I think I'd need to discuss this further with @JedWatson or somebody more familiar with how we interact with Mongoose to determine whether or not this would be easier than switching db entity references depending on the selected locale. Both approaches will require a translation layer that doesn't yet exist in the app.

I would also recommend adding a switch to explicitly add a translation to an entity because often not every content is available for every language.

I don't think this would be necessary. If the selected locale doesn't exist, simply fall back to the default? =)

ericelliott commented 8 years ago

Locale negotiation

Here are the locale negotiation priority rules I plan to implement, unless anybody has any objections. These are favored in order of precedence.

Query param, e.g., http://host:port/path?locale=en_US
If user is signed-in, user preferred locale
If cookie is present, use cookie value
browser Accept-Language header

What about geoip?

The Accept-Language header is generally the standard approach for determining locale in the absence of explicit user preference. Geoip has major downsides:

It slows down every new request, since ip location data needs to be fetched for all first-time anonymous users. These requests generally need to hit the disk or look up data in a RAM guzzling in-memory database, which will prevent Keystone from running on web host micro-instances.
IP data can be inaccurate: Not every user in the US speaks English. Not every user in Brazil speaks Portuguese.
Users visiting via proxies are likely to be served the wrong locale.

tinganho commented 8 years ago

@ericelliott I have written a middleware that handles 1, 2, 4. And the locale should use dash - and not underscore. Also there is a difference between the word locale and language. When used language the word often refers to bcp47 language tags, thus we have the accept-language header and not accept-locale header. The word locale is a more loose word than language. But in a software context the word language is more widely used than locale for describing a locale, mostly because localizing apps only involves changing the language.

https://github.com/tinganho/express-request-language

ericelliott commented 8 years ago

Your lib looks like a great start. #3 is important for testability and a last-resort manual user override. Should we add it to your lib?

Also there is a difference between the word locale and language

Yep. I'm aware of that, but most apps start inferring a full locale setting from the Accept-Language header, and combine that data with settable user preferences for things like date and currency formatting -- which seems to be exactly what we should be doing, right?

The language negotiation is just dipping our toes in the i18n waters. ;)

localizing apps only involves changing the language

I'm not sure what apps you're localizing, but I deal with a lot of ecommerce, scheduling, and location-based apps, all of which require a lot more than just setting the language. As I mentioned, inferring the language is just a first step. =)

ericelliott commented 8 years ago

I have written a middleware that handles 1, 2, 4

I see cookie and header support. I don't see user preference support (which is generally stored in a User model). Am I missing something?

tinganho commented 8 years ago

Sorry I meant only 2, 4. No user preferred option.

tinganho commented 8 years ago

Your lib looks like a great start. #3 is important for testability and a last-resort manual user override. Should we add it to your lib?

I would consider a PR on this.

ericelliott commented 8 years ago

Sounds good. I'm done working for the night, but if you don't beat me to it, I might have a PR for you tomorrow. =)

LorbusChris commented 8 years ago

@ericelliott, great your taking a swing at this! Can Smartling maybe handle XLIFF? Have a look at https://github.com/yahoo/intl-messageformat/issues/106. Not sure what still needs to be done there to make it work though.

PS: https://github.com/yahoo/react-intl/issues/213 and https://github.com/yahoo/react-intl/issues/221 may be useful, too PPS: regarding ICU<->XLIFF (and possibly other formats) https://github.com/yahoo/babel-plugin-react-intl/issues/7

ericelliott commented 8 years ago

@tinganho The PR is ready for your review. If you can jump on it fast, I'd appreciate it. I want to pull this into Keystone ASAP so we can continue progress on this issue. =)

@LorbusChris Thanks for tracking down those resources. I'll check them out! =)

ericelliott commented 8 years ago

@tinganho has done a great job with language negotiation in express-request-language. I've been working with @tinganho to merge in support for query string language setting, which will make it easy to test language options on any page.

Now looking at integrating the library with Keystone. Here's what I've drafted for preferences. Comments/questions are welcome and encouraged:

`language options`: Object (optional)

The language options is an object containing your language support preferences. Defaults to all default values and en-US as the only supported language.

`supported languages`: Array

A list of supported languages. Preferred languages come first. Language tags must comply with BCP47 standard:

language[-script][-region]

language subtag (en, zh).
script subtag (optional, e.g. Hant, Latn).
region subtag (optional, US, CN).

Examples: en, en-US and zh-Hant-TW.

`language cookie`: String (optional)

Cookie name: Name of the language cookie. It will store the current language tag of the user's session and remain until maxAge expires or until it is changed by using the browser URL overrides.

`language cookie options`: Object (optional)

Options for the language cookie. This is passed through to express. See the Express cookie options for details.

`language select url`: String (optional)

An optional URL used to change language preferences. It will redirect back to the origin URL if you send a referrer header and default to / if it don't send a referrer header.

Usage:

/languages/{language}

E.g., visiting /languages/en-US will set your language preference to en-US.

`language query name`: String (default = 'language')

You can optionally set the language on any page without a redirect using a query string. This option allows you to set the name of the query parameter that triggers the language setting. The default value is language.

E.g., if you set language query name to 'locale', adding ?locale=zh-CN to any URL path will set the language to 'zh-CN'. Use the special language value, default to unset the language preference and fall back on the default setting as if you are a first-time visitor.

`disable`

Disable the language middleware.

geloescht commented 8 years ago

I am working on a project that needs content localization more badly than i18n of the admin interface or strings. I believe apart from locale negotiation, localization of content and strings/admin interface are separate issues. Since locale negotiation has already been tackled, I think we could discuss both now. I created a wiki page with ideas on how content i18n could be implemented: https://github.com/keystonejs/keystone/wiki/Content-localization---Proposal Please add your comments, ideas, pros and cons of different solutions etc. Regarding database structure I tend to solution 1.ii, but I am not very familiar with MongoDB.

(I would open a new issue for this purpose but saw that #1450 has been closed. Maybe a member of the keystone team can decide if this is the right place to discuss content localization, #1450 should be reopened or a new issue should be filed)

tinganho commented 8 years ago

I like the language-code keys 1.a.i solution. Though I think this should be fixed upstream. MongoDB should provide a localizable field solution.

sbaechler commented 8 years ago

@geloescht I suggest the solutions for content localization be provided in framework independent libraries. This way a developer can chose which approach fits his needs and the library itself has a larger maintainer base. Each framework would then just need a small adapter for each library. Maybe the translatable configuration can be in an external file and be added to an existing model. This way it is possible to re-use models that have no translation fields. A good approach of this can be found in Django Modeltranslations for Python. Your two DB structure proposals need the current active language value at different times within the lifecycle. If all languages are stored in one model, then the current language is needed during getting and setting the values. While if the solution uses child models for translated fields, then the current language is needed for the database query. I prefer that the language is passed to the database request explicitly.

geloescht commented 8 years ago

@sbaechler I agree that making content localization an independent module would be desirable. However, I also think it is essential that keystone comes with a sensible default implementation (be it an adapter to an existing library or internal) and integration with the admin UI. Relying on application developers to integrate their translation library of choice every time they start a new keystone project would pose too big of a big barrier and lead to poor testing. If someone could point to a working, maintained content translation library for mongoose, please do. If we opt for totally pluggable content localization we would have to define an API for that purpose. I am not sure what you mean by model reuse. Do you mean that translatable fields should not be declared in the same file as the main list registration? I am not quite sure why they shouldn't, considering that parameters for the admin UI, tracking etc. are also declared there. It is true that changing the translation options for a list later might pose problems with already existing documents, but that is true for all implementations, whether they use secondary files to define translatable fields or not. That should be addressed in the implementation. I do not intent to supply a language each and every time a value in a document is accessed. Rather, the current language is saved alongside each document when queried and the correct translated fields should be saved automatically. I believe mongoose can provide that amount of flexibility without the need for additional wrapper objects. (If that is what you mean by child model. keystone has list inheritance, but that does something different)

sbaechler commented 8 years ago

@geloescht I agree that Keystone should have a default implementation for content localization. I just think it is advisable that there is a layer of abstraction between it and Keystone. Like there is between the template engine and Keystone. Imagine that in two years Mongo DB supported i18n out of the box. In that case you would just have to update the modeltranslation module when you updated your DBMS and not all of Keystone. It would be more work upfront but it made things easier down the road.

The dependency injection way of adding translations to a model is just a suggestion for an implementation approach. I think it works very well for Django applications. It's easy to add i18n to existing projects or 3rd party modules that don't have it.

geloescht commented 8 years ago

@sbaechler I see. I edited my proposal to allow for a translation service to be specified for each list separately as an alternative to a built-in service. This way we could provide a smooth upgrade path for switching translation services and even mixing of different translation schemata within the same application. Should the built-in translation service ever change in an incompatible way, a legacy service could be kept around. Please feel free to add alternative approaches to the proposal on the wiki :) https://github.com/keystonejs/keystone/wiki/Content-localization---Proposal

BTW: keystone already has a directory "schemaPlugins" with various mongoose middleware and functions. I imagine a content translation service to be one of those schema plugins. I even started on an implementation and it looks quite promising to do it that way. However those schema plugins are currently not truly pluggable.

joernroeder commented 8 years ago

I just found this issue during my research for an upcoming project which needs content localization as well. In this project every url should be translatable too (for SEO reasons) and i think that's a regular use-case. As all fields can therefore differ i propose an additional _localisationGroupId key which is shared across all localisations and defines the relationship (grouping) between the documents. I would also prefer to create a new document for each localisation which contains a simple locale key.

{_id: 'fhdsjfs', _localisationGroupId: 'hgdfgjdskfh', locale: 'en', 'field1': 'foo' }
{_id: 'kdjhfgk', _localisationGroupId: 'hgdfgjdskfh', locale: 'de', 'field1': 'bar' }

// find other translations of the current article
context.list('Article').model.findOne({ _localisationGroupId: { $not: { article.locale }})

Shared fields (fields which are untranslated) can be updated in all localisations via:

// mongo query
collection.update({ _localisationGroupId: groupId }, { $set: {untranslatedField: 'foobar' }}, { multi: true });

I would also propose to group fields in the admin UI by locale (with tabsets for example) or link to other locales similar to relationships.

geloescht commented 8 years ago

@joernroeder Thanks for sharing your ideas. I've added your proposal to the Wiki. I think it might be a little harder to implement than structures where all localization data is stored in one document (something it has in common with other approaches). Do you have an idea on how to make your approach opaque via mongoose middleware? BTW, Valeri Karpov of mongoose has accepted one of my patches for mongoose 4.5 that is necessary for my implementation to work, so hopefully I will put something out there soon :)

joernroeder commented 8 years ago

@geloescht i'll look into mongoose middleware. maybe i can come up with something helpful. I know that it's harder to implement than using a single document but this seems to be more flexible to me and should (when you clone untranslated fields into every localisation) be faster to read than "joining" documents together.

wmertens commented 8 years ago

Now, there are several separate issues here:

i18n of the admin UI, including form labels
i18n of content. I prefer being able to say per field if it should get i18n or not, instead of using multiple document copies of the same thing in different languages.
i18n of frontend views

I18n is a great bikeshedding opportunity. I propose that we implement only the minimum needed to move forward. For example, in Polish counting years goes "rok, 2 lata, 3 lata, 4 lata, 5 lat, 6 lat, …". You could implement something complex that handles all of these cases across all languages, or you could do "lat: x". If you don't worry about pluralization, you can already do a lot of good things easily, just by providing a string per language, maybe with interpolation markers.

@joernroeder we have had good success by making documents like {id: xxx, isFoo: true, someNumber: 7, title: {de: "Hallo", fr: "Bonjour", en: "Hi"}, …}. Documents are entities, and i18n is representation of some values inside the entities.

So I propose that we determine exactly what we want to to for each of the issues, create new issues and implement them.

mdorda commented 8 years ago

It's really a shame that Keystone can not display mongoose-intl in the Admin UI. That would be all I need :-)

JonathanJJudge commented 8 years ago

Hi everyone! I'm in a situation where a client really wants suburb to read as city for the location fields when managing through the Admin UI. Perhaps as part and parcel of this discussion we could add a simple configuration for overriding the subfield labels/placeholders on the Location field? I'd be happy to attempt putting together something and make a pull request for the latest release of 3, for more immediate use while 4.0 is in beta, and if that goes well, for 4.0 as well? It might be a sufficient stop-gap measure.

seidtgeist commented 8 years ago

I'm with @wmertens, it would help if there were separate issues for I18N admin, I18N content and I18N for frontend views. Each one is a big topic by itself.

My current requirement is I18N for content for which I'm going to add extra fields with locales in their field name. I found that this offers maximum control and is not that different from having something like document.field['en-US'], etc.

kelkmere commented 7 years ago

I want to ask a question to people who work at Thinkmill. What happens when one of your clients need a multilingual website?

gggllm commented 7 years ago

I am not sure if we are able to choose a language to render in adminUI in 4.0 beta, I am actually not that skilled with English. Can someone tell me something about it?

jschutijzer commented 6 years ago

What is the progress on this feature. I would like to use keystone but without multilingual feature I can't use it for customers.

mroz22 commented 6 years ago

+1

quaeched commented 6 years ago

+1

glimow commented 6 years ago

Same here, also I'd be glad to help admin translation to french if this feature comes out !

keystonejs / keystone-classic

Multiple language support #802

i18n checklist

See also

I need your help

Locale negotiation

What about geoip?

`language options`: Object (optional)

`supported languages`: Array

`language cookie`: String (optional)

`language cookie options`: Object (optional)

`language select url`: String (optional)

`language query name`: String (default = 'language')

`disable`

keystonejs / keystone-classic

Multiple language support #802

i18n checklist

See also

I need your help

Locale negotiation

What about geoip?

language options: Object (optional)

supported languages: Array

language cookie: String (optional)

language cookie options: Object (optional)

language select url: String (optional)

language query name: String (default = 'language')

disable

`language options`: Object (optional)

`supported languages`: Array

`language cookie`: String (optional)

`language cookie options`: Object (optional)

`language select url`: String (optional)

`language query name`: String (default = 'language')

`disable`