Open stasm opened 6 years ago
vladkolotvin
commented
6 years ago In some languages there are grammatical cases(https://en.wikipedia.org/wiki/Grammatical_case). This feature can help with this problem. Example in russian: Nominative case - У меня есть коровы, свиньи и овцы. (I have cows, pigs and sheep.) Genitive case - У меня нет коров, свиней и овец. (I have no cows, pigs and sheep.)
stasm
commented
6 years ago Grammatical cases are already well-supported by Fluent; see http://projectfluent.org/fluent/guide/variants.html. But you're right—these two features will synergize well :)
Pike
commented
6 years ago I think we should do this, the use-cases look good enough. Localizers' life will be hard in these cases, but less hard than with the alternative.
stasm
commented
6 years ago Thanks, @Pike. A few more examples which illustrate why it's useful to resolve the references on the localization side rather than in the code (and pass translated strings as arguments).
Let's assume a game UI which logs what the player sees:
You see a fairy.
You see an elf.-creature-fairy = fairy
-creature-elf = elf
.StartsWith = vowel
you-see =
You see { $object.StartsWith ->
[vowel] an { $object }
*[consonant] a { $object }
}.The you-see message can inspect the English-specific StartsWith attribute and choose between the correct indefinite article a or an. For conciseness, -create-fairy doesn't define the StartsWith attribute at all; the default variant in you-see will be used.
-creature-fairy = Fee
.Genus = Femininum
-creature-elf =
{
*[Nominativ] Elf
[Akkusativ] Elfen
}
.Genus = Maskulinum
you-see =
Du siehst { $object.Genus->
*[Maskulinum] einen { $object[Akkusativ] }
[Femininum] eine { $object[Akkusativ] }
[Neutrum] ein { $object[Akkusativ] }
}.The you-see message can inspect the German-specific Genus attribute and choose between the correct indefinite article for the gender of $object. The object is also correctly accorded with the verb sehen which requires the Akkusativ.
PS. The examples above don't solve capitalization (fairy and elf are always lowercase in the English translation), but I'm leaving it out on purpose. It may be solved by nested variants or a function.
zbraniecki
commented
6 years ago While working on https://bugzilla.mozilla.org/show_bug.cgi?id=1435915 I found a use case for this feature.
There's an API there which constructs a description of the application handler.
It can be a localizable term, like "Portable Document Format (PDF)" or "Video Podcast", it can be a generic description like { $extension } file, or it can be a raw string.
I handle all three scenarios using a strategy from Gaia days - the API circulates an "l10n type" object:
// a string -> l10nId
// an object -> {id: l10nId, args: l10nArgs}
// an object -> {raw: string}
{id: "applications-type-video-podcast-feed"},
{id: "applications-file-ending", args: {extension: ".mp4"}},
{raw: "Windows Video File"}, // this one comes straight from the OSThose strings are resolved in a loop and displayed in a table in Firefox Preferences in a column "type description".
Now, the trick is that there's a place in the API which separates how this string is displayed in case there are two entries with the same description.
This can happen because for example, there are two file types for "Video Podcast" or "Windows Video File".
In that case, there's a special string in Fluent:
applications-type-description-with-type = { $description } ({ $type })which is used to display Windows Video File (.mp4) separately from Windows Video File (.mpg).
With support for this UI I could use the FluentReference as $description instead of having to resolve the string with formatValue and pass it as a string.
I'm going to workaround it for now, but just thought it may be useful to know that we already encountered a use case in Firefox.
zbraniecki
commented
6 years ago This would also help with cases where a message value is used as an attribute in another element.
could be:
pane-general-title = General
pane-search-title = Search
category =
.tooltiptext = { $paneTitle }Example2:
the applications-type-description are used as values, but then placed into the XUL as <item typeDescription="..."/>. It would be useful to make it:
item-type-description =
.typeDescription = { $typeDescription }Granted, I don't know how will we store it in data-l10n-args.
zbraniecki
commented
6 years ago Aaand another use case:
language-pl = Polish
language-fr = French
region-pl = Poland
region-us = United States
locale-pattern = { $language } ($region})From https://bugzilla.mozilla.org/show_bug.cgi?id=1451450#c6:
We'll need to support VariantExpressions and AttributeExpressions on both MessageReferences and ExternalArguments:
-term[varname]
$object[varname]
-term.attr
$object.attrWhich I think is best solved by adding another level of nesting to the AST, unfortunately. Right now, -term[varname] parses as:
{
"type": "VariantExpression",
"id": {
"type": "Identifier",
"name": "-term"
},
"key": {
"type": "VariantName",
"name": "name"
}
}In order to support both MessageReferences and ExternalArguments and to be able to serialize them, I think it should rather parse as:
{
"type": "VariantExpression",
"of": {
"type": "MessageReference",
"id": {
"type": "Identifier",
"name": "-term"
}
},
"key": {
"type": "VariantName",
"name": "name"
}
}This is best visualized with the spans of $object[varname]:
$object[varname]
+----+ Identifier
+-----+ ExternalArgument
+------+ VariantName
+--------------+ VariantExpression
spookylukey
commented
6 years ago In Django we have a use case for this feature, not just as a matter of convenience - without it we wouldn't be able to generate correct translations at all. We have exactly the "Delete the selected %s items" case, but in our case, as a framework, %s is the name of a model, provided usually in English by a developer. It's therefore not known to the Django authors, but is known to the app developers. If we are translating into French, for example, the word "selected" becomes either " "sélectionnés" or "sélectionnées" depending on the gender of the substituted model name.
We would also want some way for FluentReference to provide a fallback - what happens if e.g. the application code passes in FluentReference("-creature-a-new-creature") but -creature-a-new-creature doesn't exist at all in the FTL file? We'd want to pass in FluentReference("-creature-a-new-creature", "a new creature"), and "a new creature" would be used for all variants if -creature-a-new-creature is not defined at all.
(Whether Django, with its current investment in gettext, would be able to move to fluent is another matter, but the point applies to other framework-like code, and the choices of frameworks can affect the choices of a lot of other things).
zbraniecki
commented
6 years ago I think I'm stuck with https://bugzilla.mozilla.org/show_bug.cgi?id=1435915#c15 until this lands.
stasm
commented
6 years ago Some more explanation would help :) Do you mean something like the following?
applications-action-always-ask =
.label = Always ask
applications-action-generic-label = {$menuitem.label}And then in JS:
setAttributes(
labelElement,
"applications-action-generic-label",
{
menuitem: new FluentReference(menuitemElement.getAttribute("data-l10n-id")),
}
);It's still an open question for me whether we should allow dynamic reference to messages. In fact, I'd prefer to start by allowing dynamic references to terms only. I have concerns about dynamic reference being abused in scenarios where they're not about grammar. In bug 1435915 comment 16 I suggested a slightly more verbose alternative which will fix the problem outline in the bug. IIUC, the real fix would be to encapsulate the variable shape of the translation with a WebComponent.
sn-o-w
commented
5 years ago https://pontoon.mozilla.org/ro/common-voice/cross-locale.ftl/?string=177874
In this case, no matter what {$lang} will display, I need lower-case for first letter, no matter what locale name will be. This is how Romanian language works.
For example it should display "Mulțumim pentru interesul de a contribui la română" ("Thank you for your interest in contributing to Romanian").
If there could be a way to force this by sintax or to define specific new rules for Romanian, that would be great!
And maybe an option for uppercasing too, not only lowercasing. It could be useful perhaps for other languages.
aphillips
commented
5 years ago @cristisilaghi Note that CLDR provides different contextual strings for "standalone" vs. "in context" display of strings like language, region, time zone name (or other values, such as month names, etc.).
zbraniecki
commented
5 years ago Another example where this would save the day: https://phabricator.services.mozilla.com/D15762
we have two long lists of language and region names:
region-name-us = United States
region-name-pl = Poland
region-name-de = Germanylanguage-name-pl = Polish
language-name-de = German
language-name-en = Englishand a single connector:
locale-display-name = { $lang } ({ $region })with dynamic refs I could do:
document.l10n.setAttributes(menuitem, "locale-display-name", {
lang: FluentMessage(`language-name-${langCode}`),
region: FluentMessage(`region-name-${regionCode}`)
});and have it properly retranslate on locale change and such.
alabamenhu
commented
5 years ago I have been working a P6 implementation of Fluent while porting over a text adventure game and I definitely agree dynamic term references are necessary, and shouldn't be too hard to implement.
The initial proposed syntax of just using $foo[bar] would cause ambiguous entries for languages that don't need case/etc) and don't have variants as both a variable and a variable-powered term reference would look $foo(When translating from say, an en.ftl file, it also let's the localizer know if they need to try to creatively word to avoid case/etc issues, or if they know it's going to be a term, so they can then add on case information).
To me the better syntax would be -$variable, as $ is not a valid identifier character, so $variable cannot be confused by parsers as a term. In effective, it would be a variable term reference that just simultaneously performs the actions of a variable and a message reference (and hence -$).
I'm haven't dug deep into the internals of other implementations, but for the way I wrote the P6 implementation, it took me about 5 minutes to add support for -$ and a VariableTermReference class.
Here's the FTL file I used and the results:
-dog =
{ $style ->
*[normal ] dog
[diminutive ] puppy
[diminutive-redup] puppy dog
}
-cat =
{ $style ->
*[normal ] cat
[diminutive ] kitten
[diminutive-redup] kitty cat
}
cute = Wow, that { -$animal(style: "diminutive") } is so cute!
stupidcute = OMG, that { -$animal(style: "diminutive-redup") } is like so amazeballs cute!
handsome = That's a handsome { -$animal(style: "normal") }.To call it, nothing changes from a regular variable:
say localized("handsome", :animal<dog>);
say localized("handsome", :animal<cat>);
say localized("cute", :animal<dog>);
say localized("cute", :animal<cat>);
say localized("stupidcute", :animal<dog>);
say localized("stupidcute", :animal<cat>);And the results are fairly intuitive:
That's a handsome dog.
That's a handsome cat.
Wow, that puppy is so cute!
Wow, that kitten is so cute!
OMG, that puppy dog is like so amazeballs cute!
OMG, that kitty cat is like so amazeballs cute!
It's currently posted to the repository (but not listed as a release) if anyone wants to play around with it.
zbraniecki
commented
4 years ago Another use case - https://phabricator.services.mozilla.com/D50443
Pike
commented
4 years ago I've come full circle on this one.
I still like the API and its intent, but the implications on the engineering and localization processes are tough.
I can see dynamic references be an interesting option for an ecosystem where the developers land all localization as part of their feature development, including appropriate testing and tests.
For other environments, the questions below just don't come up with good answers to me.
Background: Dynamic references basically re-establish string concatenation, and with terms, add bidirectional dependencies. Those dependencies go between all messages that include dynamic references, and all referenced terms. These dependencies are also strongly language dependent.
Starting with testing, you need to ensure that a linguist/translator has reviewed the results. Which means you need a test plan for each language, covering any grammatical combination of phrases and terms. So, you need a linguist to develop the test plan, and a translator to review. For each target language.
For creating localizations, we know that l10n tools are really not mastering string concatenations. Or including test plans in the UI. Assuming one has the test plans from above, you'd need an l10n/fluent engineer to adjust the implementations for each language. Even for the limited complexity of Terms in Firefox, most of them are done with flod's help.
Another problem arises with partial translations. Falling back to Terms in a foreign language is probably not what you want. At least, it might make the problem harder rather than easier, as you not only need to deal with attributes and variants of your own language, but also with those of others.
And then I wonder how to do change management for this. Say you have 5 player characters, and 3 monsters. And 15 strings using term references into each. Add a monster. I see tears.
I'm starting to think that the energy that needs to go into maintaining this would often be greater than the energy of just creating 200 strings with a script. Conceptually doing Term references in your source language, but not in the actual translation process.
Again, I can see ecosystems where these challenges are easily met. Like, if you're a game dev shop, and you need to have linguistic and cultural experts for all your target languages in the same office as your devs anyway.
For mozilla, though, I don't see us being able to prove this feature. Which makes me think that we shouldn't be the ones that drive this.
zbraniecki
commented
4 years ago How does this differ from:
let monster = await document.l10n.formatValue(selectedMonsterId);
document.l10n.setAttributes(msgElement, messageId, { monster });?
It seems to me that this is what people will do in absence of dynamic references and it has all the cons you listed and none of the pros. Additional cons of this approach are:
spookylukey
commented
4 years ago I'm starting to think that the energy that needs to go into maintaining this would often be greater than the energy of just creating 200 strings with a script. Conceptually doing Term references in your source language, but not in the actual translation process.
For some use cases - like the ones I mentioned for Django, which will apply to other frameworks - this solution is simply not an option, because we don't know the strings ahead of time, they are supplied by other developers. We'd be left with the kind of 'solution' that @zbraniecki has, which leaves you with broken translations for many uses cases (inability to deal with case/gender agreement etc.).
Pike
commented
4 years ago So, Zibi's example is actually interesting in two ways:
Firstly, it emulates message references. And message references are easy, and also kinda pointless as they're completely atomic.
Secondly, it adds fallback for missing message references. In his code example, messages are resolved on the Localization abstraction instead of the Bundle. Which solves a lot of problems we have with message references right now. Even just static ones. I'd love to discuss how message references work as part of the resolver standardization. But I'm also realistic about not getting a fully sync and fully async resolver implemented for all impls that want both. Neither of js, python, or rust have generic sync/async programming, right?
To Luke's comment: Terms are effectively language-dependent APIs. Messages referencing terms need to know the API, and all terms for that use need to implement the same API. With static term references, that's already nasty. With dynamic term references, it's an order of magnitude worse.
And when you talk about different software packages ... .
Say, the German team of the django localizers decides to change the Term API for contrib.admin. Now, all generic apps with models need to update their l10n, and all custom templates that use model names need to update. And at best, you get release notes to communicate that. I guess.
Also, to clarify, I'm just saying that Mozilla isn't the right org to drive this. That doesn't mean that we shouldn't build the Fluent ecosystem such that someone else can give this a shot. Their task is going to be to figure out these things, beyond writing down APIs and syntax.
zbraniecki
commented
4 years ago Are you saying that for a scenario like https://github.com/projectfluent/fluent/issues/80#issuecomment-451618060 we should generate lang x region combinations into an FTL file for Mozilla needs?
Pike
commented
4 years ago I would just go for computed values and retranslations.
alabamenhu
commented
4 years ago It's true that allowing dynamic references would put more work on the translator. But that's the eternal balance that must be played: more work for the developer to create hundreds of nearly identical strings but for swapping out a word or two, or more work for the translator to understand the tech side.
But I think the reality is that most strings are fairly basic, and only a handful would require the level of detail that would make a localizer need to look up some syntax. But that's already somewhat expected, after all, given how Fluent is designed to help us move away from "Number of files: X" to "No files" but "1 file" or "2 files", etc. — that work befalls the translator. Rare would be an application that needs extremely complex logic requiring dynamic message references, but better to make it possible than preclude it entirely.
I can completely understand Mozilla not wanting to be the driving force if it doesn't have an internal use case (althpugh it sounds like it does), but defining a standard syntax and providing a baseline (even if suboptimal) implementation would do well to further the adoption outside of Mozilla and prevent splintering of the format.
zbraniecki
commented
4 years ago Another example of where this would be helpful in Firefox - https://bugzilla.mozilla.org/show_bug.cgi?id=1642725
Without that, formatValue is required to format the message before passing the result as an argument to setAttributes.
The issue with that is three fold:
The last is an example of the first in this case - the developer causes all setAttributes to be called after awaiten formatValues is resolved which pushes back the l10n microtask in which they all get translated significantly, just to resolve the microtask with formatValues.
Having Dynamic References would make this code clean, intentional, and easy to optimize the DOM bindings around.
zbraniecki
commented
4 years ago Another potential case - https://phabricator.services.mozilla.com/D80944
In this case, we need to evaluate how the brand name of the product affect the structure of the sentence. It may not be possible to easily place the brand in nominative form, or it may be that we'll have to denote that the argument is in nominative form and ask localizers to adapt the sentence to it.
A scenario I imagine might be the most flexible is:
For locales where the sentence doesn't depend on any aspect of the variable, use dynamic references:
browser-ie = Internet Explorer
browser-chrome = Google Chrome
browser-safari = Safari
autocomplete-import-logins = Import your login from { MESSAGES($browserMessageId) }For locales where it does,
autocomplete-import-logins = { $browserCode ->
[chrome] Import your login from Chrome
*[other] Import your login from { MESSAGES($browserMessageId) }This would allow localizers to adapt sentences which they need and leave the generic form (potentially imperfect) as a other fallback.
stasm
commented
4 years ago Another potential case - https://phabricator.services.mozilla.com/D80944
The solution this patch settled on is also what I think is the best localization practice for a small number of variants:
autocomplete-import-logins-from-chrome = Import your login from Google Chrome
autocomplete-import-logins-from-ie = Import your login from Internet Explorer
autocomplete-import-logins-from-safari = Import your login from Safari
# etc.This has the best chance of producing translation of good quality. The localizers have full context in each string, and are also free to introduce any changes to spelling, declension, and others, as they see fit, because each string is independent.
alerque
commented
4 years ago The solution this patch settled on is also what I think is the best localization practice
If that's the best practice, how is Fluent better for this use case than a YAML file with key/value pairs? Sure breaking out every possible variation into a separate key gives you absolute control, but then the burden of translators goes way up (and translation editing tooling gets tasked with trying to lighten the load through suggestions from similar strings etc. which turns into a mess when you start updating anything).
stasm
commented
4 years ago That's a good question, thanks! I think the notion of the localizer's control is key. A simple key/value pair store takes away this control when we consider plurals, genders, or some forms of declensions. If the source language (often: English) doesn't support a grammatical feature required by the target language, the possibility of creating a well-sounding translation is limited.
In the Import your login from… example, however, the reason for the variation is not language-specific: the list of supported browsers is known ahead of time and constant across languages. In this case, I think separate messages offer the most control to localizers, again. If a language requires declension or a different article of some browser names, the localizer can modify the relevant string inline. Does that answer your question?
m-aciek
commented
2 years ago Hi, another use case example. It comes from Django. The snippet provides correct Polish translations to sentences consisting of model objects' count and the model's name, like "1 user" or "5 groups" in English. In Polish one of grammatical genders – "masculine personal" (męskoosobowy) – is an exception from others, and requires genitive instead of nominative in plural form for one of the plural categories.
main.pyalerque
commented
2 years ago Just a heads up, Linguist has Fluent support now, so you can mark your code blocks for highlighting:
```fluent
example = foo
```
Ygg01
commented
1 year ago So any news on implementing this? Or should we just implement our own?
eemeli
commented
1 year ago This is not currently being worked on. In large part progress here is blocked due to the uncertainty of how or whether MessageFormat 2 will be able to support dynamic message references, and not wanting to introduce new Fluent features that may be challenging to make compatible with MF2.
Shelim
commented
1 year ago @eemeli Even as optional feature, to be able to be only explicitly enabled in given project (with red-alert-style warnings that it may break compatibility with MF2)? It is a highly no-go for dynamic content (as stated above - I am actually investigating Fluent for a game and the Heroes III example is my exact deal-breaker with numbers. Paying translator to translate 10,000 message variants is an awful idea to translate a single line...)
Shelim
commented
1 year ago Status update: I was actually able to workaround my way into this. Relevant playground [Polish, following Heroes III example]: https://projectfluent.org/play/?id=dbf872642497ea2b98efe2afa7585dc1
Steps:
FluentMessage of the target creature ( for example "dragons")FluentMessage for being killed by with the populated attributes collectionIt is actually fairly easy to include this in application-side code and is obvious enough for the translators to keep this as viable solution
zbraniecki
commented
1 year ago The downside of your approach, which may not be really that relevant for your use case, is that you resolve your first message and second separately. It means that any locale change requires both calls to be re-run for the new locale, which is quirky (at least in DOM scenario).
Main value of dynamic references is that it folds this sequence into a single API call between L10n system and the caller securing locale consistency.
It's a bit analogous to as if instead of having You have { $count } emails have the number resolved in MF2, we'd ask developers to format the number, and pass formatted number as a string to MF2 keeping track of locale alignment between the two. We could, but it'd clearly be a bad architecture.
Shelim
commented
1 year ago @zbraniecki This is why I call this a "workaround" 😄 But since it is obviously hitting a wall for a proper implementation (and there is no other good quality alternative) I will take this one. It suit my needs good enough and I do not use DOM at all in desired game.
It is sometimes desired to parametrize message references in placeables. In this issue I'd like to propose a new argument type, extending
FluentTypewhich could be used to programmatically pass message references as arguments to messages.Problem Statement
Redundancy is considered good for localization. It allows localizers to tailor the wording and the grammar of the translation of each particular case. Also see Fluent Good Practices.
In general, the pattern of having one message per item is preferred over factoring the action out to its own message (
Delete This { $item }) and passing the translateditemin some way.In some cases, however, this pattern doesn't scale well.
Consider this example from Firefox (source):
Or the use-case @cruelbob gives in https://github.com/projectfluent/fluent/issues/79#issuecomment-356195985:
One of my favorite games, Heroes of Might and Magic III, pits armies consisting of over 140 different unit types in battles against each other. After every move, the battle log reads:
Or:
If we wanted to avoid concatenation of sentences (two sentences per creature: one for
do X damageand one forX creatures perish), we'd end up with 141² = 19,881 different permutations of creature pairs.This doesn't scale well.
Proposed Solution
Introducing some redundancy should still be preferred for small sets of items. For large sets leading to lots and lots of permutations, it should be possible to parametrize the translation of placeables.
I'll use the example of HoMM3 because the other two also require the List Formatting feature to make sense.
I'd like to make it possible to pass external arguments which resolve to message references. Given the following FTL:
…both
$attacker_nameand$defender_namewould be arguments of typeFluentReference(extendingFluentType; same asFluentNumberandFluentDateTime). The developer would pass them like so:This change mostly requires additions to the
MessageContextresolution logic. Syntax-wise, theVariantExpressionand theAttributeExpressionshould be changed to accept both message identifiers as well as external arguments as parent objects (like in the$attacker_name[singular]example above).Open Questions
Sign-offs
(toggle)
##### @Pike - [ ] I support this. - [ ] I don't care. - [ ] I object this. ##### @stasm - [ ] I support this. - [ ] I don't care. - [ ] I object this. ##### @zbraniecki - [ ] I support this. - [ ] I don't care. - [ ] I object this.Also CC @flodolo.