Message context for Yii::t()

[Deele]

I feel that Yii::t() lacks ability to provide context of textual value and when translations start to count in couple hundreds, it starts to be quite hard to understand what was meant with translation.

If we look at popular translation systems like "Crowdin" (example), they allow to provide message context to support translators.

Currently, only way to provide some helpful information using Yii::t() is to write something in "key" area, that in turn defies purpose of "source language", for example, if you write a message "Propose (button in issue tasklist table header)" until I change that translation to "Propose", that message still be visible.

I am guessing, that there is performance issue with long keys, and length limitations does not allow long context descriptions.

For backwards compatability, I would like to suggest 5th optional attribute to Yii::t($category, $message, $params = [], $language = null, $context = null) (but this is not very convenient) that would allow to provide some sort of context description.

Maybe, we could convert Yii::t() to allow passing in first attribute array of attributes, like Yii::t(['category' => 'message.category', 'message' => 'message text', 'params' => [], 'language' => null, 'context' => null]) or even allow shorthand for category and message with numeric keys Yii::t(['message.category', 'message text', 'params' => [], 'language' => null, 'context' => null]).

With \yii\i18n\DbMessageSource, they would be stored in separate table source_message__context, where we would store source message ID and context description with 255 characters long string.

With \yii\i18n\PhpMessageSource, they would be stored in separate file, right next to current message files, in {category}_context.php format, where we would have associative array with source message category as a key, and array with context descriptions, as a value.

I have no experience with \yii\i18n\GettextMessageSource, but I see in their documentation, that they already have an option to provide context descriptions. But I would like to see some suggestions for this solution.

This feature would allow to display all of those context description together with message in interface, to help translator to translate messages and understand context of it.

[samdark]

Getext context is what we know as a category, not an example of usage: https://www.gnu.org/software/gettext/manual/html_node/Contexts.html

[klimov-paul]

I am against such feature. Placing some descriptive information around any call of Yii::t() sounds like overkill to me.

Besides how we can control same pair of 'category' and 'message' will have same 'context' at all its invocations? For example:

Yii::t('app', 'Some message', [], 'en', 'Some context');
// ...
Yii::t('app', 'Some message', [], 'en', 'Another context!!!');

Each translation message has unique key, which is a combination of category and message. If you wish to add some description to them you can use this pair and your own custom wrapper.

[samdark]

Agree.

[Deele]

@klimov-paul I don't see arguments about such approach being overkill. I already explained how to control/store them. Each message can have multiple contexts as in your example. Nothing changes regarding identifiers.

[samdark]

@Deele

1. Not all formats we support allowing for such feature. GetText does not.
2. There's category to differentiate contexts.
3. You have to preview your translation somehow anyway. It's not possible to translate 100% perfectly w/o checking actual UI usage because of many factors: context which could not be effectively described in text, UI specifics etc.

[Renkas]

1. GetText supports comments: https://www.gnu.org/software/gettext/manual/html_node/PO-Files.html
2. it's not very descriptive. A freetext comment can be usefull sometimes.
3. Of course - but translators don't have that possibility usually (at the time of translation).

[samdark]

OK. If comments are supported in GetText we may introduce it.

[samdark]

Still, I don't think these descriptions are good to be placed right in the code.

[klimov-paul]

I don't see arguments about such approach being overkill.

For me internal program code should not contain such information since it does not influence the program logic. You propose to make source code, which will be executed on each(!) program run much heaiver to support translation functionality, which is performed relatively rare.

Yii::t() function serves the functioality purpose, while yii message command is a service. I do not like the idea to mix up functionality with the service.

Each message can have multiple contexts as in your example

And how it should look like in the end? How it should be stored? What it should mean for the interpreter who will work with translations?

One more thing. What you are planning to do in case your interface for the translations management should also provide i18n feature? This will mean that your 'context' should be translated as well. Otherwise it make not much sense then a plain message ID.

[klimov-paul]

GetText supports comments:

GetText instroduce sperated file per each language. So you mean 'context' may vary per each language in your case? As far I can see 'context' is something universal through all language translations. Saving context inside GetText will just cause a lot of data duplication.

[Renkas]

Don't confuse comments with context/category.

[SilverFire]

With \yii\i18n\PhpMessageSource, they would be stored in separate file, right next to current message files, in {category}_context.php format, where we would have associative array with source message category as a key, and array with context descriptions, as a value.

I read this and can't find points why the aforementioned is worth than creating a separate dictionary for a specific context. I see the following cons:

• New entity
• More code to support
• More complexity to Yii::t API (passing 4-th argument after two optional)
• More complexity to messages resolving
• Need to re-write yii message, as @klimov-paul said in his last message

Therefore I'm standing against this feature request.

[Renkas]

xgettext utility works like so:

Comments (starting with ///) placed directly before strings thus marked are made available as hints to translators by helper programs. https://en.wikipedia.org/wiki/Gettext#Programming

I see the problem having comment in a method call. Couldn't the message parser support some kind of comments markup?

/// Some hint for translators here
echo Yii::t('app', 'Awesome string');

And for inline usage maybe the parser could be smart enough to recognise comment that is inserted immediately after method call?

echo '<b>' . Yii::t('app', 'Awesome string')/* Some hint for translators here */ . '</b>';

Of course the parser should then be able to work when there are two method calls and hints on same line (example for test case ;) ):

echo '<b>' . Yii::t('app', 'Awesome string')/* Some hint for translators here */ . '</b> - <b>' . Yii::t('app', 'Another awesome string')/* Hint for the other string */ . '</b>';

[Renkas]

@SilverFire so you dislike the idea because it means someone needs to re-write yii message? Or why did you down vote without any comments? All other points from your earlier post has been addressed in this solution.

Anyway I don't really like the opening and closing of this thread in a short amount of time without waiting for answers to open questions. And down voting from Yii core developer without giving a reason after all the issues have been addressed is a very odd move.

[rob006]

WordPress uses such approach. https://developer.wordpress.org/themes/functionality/internationalization/#descriptions https://develop.svn.wordpress.org/trunk/tools/i18n/extract.php

[SilverFire]

Or why did you down vote without any comments

I was disturbed while writing a comment. Then found, that I've confused translators comments and context. :unamused: @Renkas sorry

[klimov-paul]

Anyway I don't really like the opening and closing of this thread in a short amount of time without waiting for answers to open questions. And down voting from Yii core developer without giving a reason after all the issues have been addressed is a very odd move.

We have provided all necessary explanations. We consider this feature is not worth the overcomlication of the code (both yii2 core internal and possible actual project one). We can not accept such changes into the main codebase in near future.

However, you can implement this feature as a separated extension outside the yiisoft scope and use it as you pleased.

[Renkas]

@klimov-paul I don't think you read the last posts here ... There's exactly 0 changes to Yii core code. Only thing that needs changing is message extraction and some documentation

[klimov-paul]

So you consider yii\console\controllers\MessageController is not a part of Yii2???

[Renkas]

I haven't said that. If it would not be a part of Yii then there would be no point for this thread would it?

But it's not part of core code as you said - core code would run perfectly fine without it. It's an utility - and quite useful one (at least for me). Support for message hints (should be the proper name for this feature I think?) is currently missing from it. And I don't really see why it could not be a part of the default functionality.

Right now it seems you just try to defend the decision on why the issue is closed and are not even thinking about the new proposal. That's not a very productive way of improving anything (which is what software development is all about I think?).

Anyway I'm not going to debate you on this anymore. If Yii team has made up it's mind then so be it. If I really need it I'll implement it myself - just seems like a thing that would probably be useful for lot's of developers.

[kidol]

@Renkas Can't you use actual message as descriptive context? What I mean is, set sourceLanguage to some dummy value, then run message command and translate to language.

'Welcome {user} # some context' => "Welcome {user}"`

[Renkas]

Of course this is a possible hack and as far as I have seen from forum threads etc some people use it. But it's still kind of a hack.

My projects are all using proper base language as strings as I'm not a fan of this dummy language solution. But that's just my preference ...

[samdark]

I'd like to discuss it a bit more after some thinking. Adding more functionality to message parser isn't a big problem. The problem is not to complicate default usage which doesn't have this context.

Using comments for the purpose looks a bit like hacking to me and, as @klimov-paul said, it adds load to APC/OpCache requiring more memory to store these.

@Deele, @Renkas do you want these comments to appear strictly in the code or you just want them to be there to be filled later?

[Renkas]

Me personally would like the possibility to write some hints/explanations to the translators when needed. In my use case I use parser to generate gettext files that are then synced to external translation service and then synced back to my application (overwriting the parsed .po files).

So what I want is a mechanism for the message parser to get these message hints from my application source code and save it to the generated .po file - which will be synced to the external service and the hints will be available there for translators.

• These hints are in itself an exception - would only use when really needed - not everywhere.
• One problem raised before was if the same message has multiple different hints/comments in different places in code.
  • My solution would be to append all of them (dividing them clearly of course).
  • When same string used in multiple places has hint defined only in one place then only that would be used - and there would be no problem
• Load on APC/OpCache part seems not to be a real problem. In PHP source you have way more comments anyway. If you mean caching .po files then yes there would be some extra lines - but I think that's quite marginal.

Of course there are other message sources to think about. In PHP message source I think they could be used as PHP comments as well? And in DB message source maybe add a extra column?

[Deele]

The message and category are key fields with limited length and other limitations. I don't want to disturb this behavior, that is why putting more information in category or the message does not really work. I want separate field, that is not keyed and indexed and could be provided within code.

I agree to @klimov-paul that there is a problem of overhead because of descriptions that are parsed in every code execution. That could be handled with use of comments as @Renkas proposed, but that could become ugly for multiple inline Yii::t() calls and even more ugly for parser to recognize.

If it is not possible to provide context description within code itself, then this feature should be handled separately altogether without changes to Yii::t() with separate table/model/solution (using message/category as the key) and this specific discussion is over.

Another idea is to gather and make available other kind of message context information - where exactly in code that specific Yii::t() is called, so that in interface, it would look like list of file names, class names, method names, line numbers and other information (to be able to create a URL to create clickable reference to repository, if application has one), that could be used by technically advanced translators, to use source code as a guide. This is not really user friendly approach, but could be much easier to implement and would not require any changes to Yii::t() syntax.

[samdark]

I'm against expanding method signature but comments approach looks at least acceptable:

1. It won't affect code execution much if not used everywhere extensively (number of comments affects how much is saved into memory from APC/OpCache).
2. It will be backwards compatible.
3. It would require to change only messages table and the message console command.

Sounds like a plan to me.

@Renkas, @Deele, is that what you want?

[samdark]

@klimov-paul, @SilverFire the approach above seems to not complicate framework much except console command message parser.

[klimov-paul]

The main problem here is a extracted context should be saved somewhere. Each particular message storage should provide a way to store it.

For the GetText its internal comments feature is proposed to be used. For the PHP and DB it should be defined.

As we have already found out, each message may have several contexts. Thus context should be stored appropriately. For example: for DB there should be an extra table, which will reference as 'many-to-one' to the message table.

[Renkas]

@samdark yes that would cover my needs

[kidol]

How will it help a human translator if a unique message has several (different) comments attached to it? I don't get it.

[samdark]

@kidol indeed. @Renkas ?

[Renkas]

@kidol so what is the problem with it?

I think i already covered it in:

One problem raised before was if the same message has multiple different hints/comments in different places in code. My solution would be to append all of them (dividing them clearly of course). When same string used in multiple places has hint defined only in one place then only that would be used - and there would be no problem

I don't see what is the problem with it? You think that same message in same category could have conflicting hint or what? I dont think that is something that is in the scope of this change. I think that is something that should be handled at the app developers level not in framework.

[kidol]

@Renkas Let's say I do:

 Yii::t('app', "unique message", "context message 1");
...
 Yii::t('app', "unique message", "context message 2");

What's the point in storing context message 2 here? How will it help translator??

This would make sense if context was part of the key, so that translator may choose to do two different translations in target language for a unique message in source language, but that's not what this proposal is about as I understand.

So I see no benefit for translator and thus no benefit to add extra complexity, ie:

For example: for DB there should be an extra table, which will reference as 'many-to-one' to the message table.

[Deele]

@kidol That is the point - same message can have slightly different contexts, but still, same meaning.

To explain why message context is needed at all...

Approach for developer would be:

• When there is a new message, he describes in one or two sentences what is the purpose of this text creating new context.
• When message is reused, developer checks original message context for similarities (to determine if the same message can be used)
  • If message is used in the same way as original, the next usage does not require him to add the same context another time
  • If message context is quite similar, but some clarification is required, developer adds a new context description

For example, the message "call" could be used as "call (the short version of longer "Phone call" in the table header)", "call (of function)" or "(the event of) call". Even thou, in this case, message categories will be different (hopefully), each message will have its own description of context, that helps not only developer to choose whether or not he should create new unique message or reuse existing, but also translator, to choose the best way to translate that text message to another language or request developers to separate those messages in two separate ones, because of invalid usage.

[Deele]

@samdark I already expressed my doubts about comment usage. How would that look like? What exactly would be syntax?

Right next to the rightmost parenthesis

echo '<b>' . Yii::t('app', 'Awesome string')/* Some hint for translators here */ . '</b> - <b>' . Yii::t('app', 'Another awesome string')/* Hint for the other string */ . '</b>';
<?php
echo '<b>' .
    Yii::t(
        'app',
        'Awesome string'
    )/* Some hint for translators here */ .
    '</b> - <b>' .
    Yii::t(
        'app',
        'Another awesome string'
    )/* Hint for the other string */ .
'</b>';

At the end of function call

echo '<b>' . Yii::t('app', 'Awesome string'/* Some hint for translators here */) . '</b> - <b>' . Yii::t('app', 'Another awesome string'/* Hint for the other string */) . '</b>';
<?php
echo '<b>' .
    Yii::t(
        'app',
        'Awesome string'
        /* Some hint for translators here */
    ) .
    '</b> - <b>' .
    Yii::t(
        'app',
        'Another awesome string'
        /* Hint for the other string */
    ) .
'</b>';

What about spacing before/after comment block? What about \n, \r \t and leading * usage (for PHPDocumentator styled descriptions)? What would be limitations to such messages?

[samdark]

Yes, it's going to get messy but these comments aren't part of the code and I'd like to keep these out of it.

[samdark]

It seems we have no good way to implement it (method signature change isn't desired, comments have issues). At the same time there are categories so contexts could be separated.

[Renkas]

What kind of issues there are with comments?

What's wrong with this implementation? https://github.com/yiisoft/yii2/issues/12182#issuecomment-245933148

[samdark]

What kind of issues there are with comments?

https://github.com/yiisoft/yii2/issues/12182#issuecomment-246177890

[rob006]

@samdark Why not support only basic syntax and ignore all weird variations? All these examples could be converted to:

/// Some hint for translators here
echo Yii::t('app', 'Awesome string');

[Renkas]

I don't see any issues there.

1. Any comment that is on the line before Yii::t call is a contextual comment
2. Any comment that is after Yii::t call is a contextual comment (there could be whitespace between method call and comment.
3. (optional) Any comment that is after Yii::t call lines semicolon is a contextual comment

Or just go the easiest route and implement only basic syntax like @rob006 suggested. It would still be much better than no support at all.

[samdark]

OK. That actually makes sense. We won't work on it ourselves though since it's very low priority so you may submit a pull request if you need it.

[bizley]

Any plans for these hints to appear in the PhpMessageSource files? If so how to add them? Like this?

'original' => 'translated', // copied hint here

Will modified in-code hint update the source file as well?

In case the content is needed we usually use label approach like

Yii::t('app', 'user.registration.username')

which with forceTranslation set to true works perfectly fine. And the label can be hint of course.
And no additional work is needed.

[samdark]

Like this?

Yes.

Will modified in-code hint update the source file as well?

Ideally yes.

which with forceTranslation set to true works perfectly fine. And the label can be hint of course. And no additional work is needed.

Yes, it's a good way to solve the issue and it's in the guide.

[samdark]

Post from Trello team mentioning similar context messages: http://tech.trello.com/ios-i18n-designing-for-translators/

[lynicidn]

зачем контекст передавать, если он указывается в конфиге, юзайте разные категории. если и хотите запилить передачу контекста, тогда придется убирать translations из i18n, а без них теряется смысл переменных, такие как fileMap для phpsource

[samdark]

@lynicidn the discussion is in English so please stick to using it. Thanks.

[lynicidn]

@samdark ok, sorry, no problem, wait pr, may be i have wrong opinion

[PowerGamer1]

Here is an idea how to solve the problem of passing a hint for translator on how to translate the message.

Yii::t('app', 'Some message text'); // message without hint
Yii::t('app', '[[When translating this message take into account the following ...]]Another message text');  // message with hint

In the example above the [[ and ]] are some user CONFIGURABLE delimiters that mark the beginning and end of the hint for translator. For simplicity, the hint for translator must always appear at the beginning of the message and cannot contain the delimiters inside itself. Of course, the message may contain no hints.

Yii2 engine code impact:

• No changes to message extraction machinery or the prototype of Yii::t().
• The Yii:t() needs to be modified to check if the message starts with a hint and if so remove it completely before further processing the message (the way it processes it now).
• Add the ability for the user to somewhere configure the delimiters for the hint to be used in the message.

Anyone sees any drawbacks in this approach other than (slight?) performance impact?

[samdark]

You're mixing message itself with a hint in a single string. Usually that's not a good thing to do and in this case, while better than previous options, it isn't as well...