Inconsistent js-translation.json generation between versions of libicu (used by php intl)

Luwdo commented 4 years ago

Preconditions (*)

Fresh Magento 2.3.5
Load application source into a machine with an older version of libicu (for my test I used centos/php-73-centos7)
Load same application source into machine running newer version of libicu (for my test I used php:7.3-cli-stretch)

Steps to reproduce (*)

run static:content:deploy in both machines and extract the source for comparison
and do a diff -qr centos_compiled_folder ubunutu_compiled_folder

Expected result (*)

pub/static and var/view_preprocessed should be identical between environments

Actual result (*)

the following js-translation.json files differ

Files centos7/pub/static/adminhtml/Magento/backend/en_US/js-translation.json and ubuntu/pub/static/adminhtml/Magento/backend/en_US/js-translation.json differ
Files centos7/pub/static/frontend/Magento/blank/en_US/js-translation.json and ubuntu/pub/static/frontend/Magento/blank/en_US/js-translation.json differ
Files centos7/pub/static/frontend/Magento/luma/en_US/js-translation.json and ubuntu/pub/static/frontend/Magento/luma/en_US/js-translation.json differ
Files centos7/var/view_preprocessed/pub/static/adminhtml/Magento/backend/en_US/js-translation.json and ubuntu/var/view_preprocessed/pub/static/adminhtml/Magento/backend/en_US/js-translation.json differ
Files centos7/var/view_preprocessed/pub/static/frontend/Magento/blank/en_US/js-translation.json and ubuntu/var/view_preprocessed/pub/static/frontend/Magento/blank/en_US/js-translation.json differ
Files centos7/var/view_preprocessed/pub/static/frontend/Magento/luma/en_US/js-translation.json and ubuntu/var/view_preprocessed/pub/static/frontend/Magento/luma/en_US/js-translation.json differ

every file differs in the exact same way

diff --git a/centos7/var/view_preprocessed/pub/static/frontend/Magento/luma/en_US/js-translation.json b/ubuntu/var/view_preprocessed/pub/static/frontend/Magento/luma/en_US/js-translation.json
index 29d55c8..9ac1e6b 100644
--- a/centos7/var/view_preprocessed/pub/static/frontend/Magento/luma/en_US/js-translation.json 
+++ b/ubuntu/var/view_preprocessed/pub/static/frontend/Magento/luma/en_US/js-translation.json
@@ -1 +1 @@
-{"Securely login into our website using your existing Amazon details.":"Securely login to our website using your existing Amazon details.","There are {placeholder} messages requires your attention.":"There are {0} messages requires your attention."}
\ No newline at end of file
+{"Securely login into our website using your existing Amazon details.":"Securely login to our website using your existing Amazon details."}
\ No newline at end of file

This new phrase is being read from: vendor/magento/module-ui/view/base/web/js/grid/editing/editor.js:525

The reason its showing up is because the translate logic is getting a different value out of the translate call then what it passed in, even when no arguments are passed and no translations are actually done.

I have determined how and why this happens: in vendor/magento/module-translation/Model/Js/DataProvider.php calls $this->translate->render() this calls vendor/magento/framework/Phrase/Renderer/Composite.php which calls the following renders in this order:

Magento\Framework\Phrase\Renderer\Composite
Magento\Framework\Phrase\Renderer\Translate
Magento\Framework\Phrase\Renderer\MessageFormatter
Magento\Framework\Phrase\Renderer\Placeholder
Magento\Framework\Phrase\Renderer\Inline

there is a debug output from this particular phase being run on the centos system:

phrases: Array
(
    [0] => You have successfully saved your edits.
    [1] => There are {placeholder} messages requires your attention.
)

class: Magento\Framework\Phrase\Renderer\Composite
translated phrase: You have successfully saved your edits.
render class: Magento\Framework\Phrase\Renderer\Translate
translation result: Array
(
    [0] => There are {placeholder} messages requires your attention.
    [1] => There are {placeholder} messages requires your attention.
)

formater locality: en_US
render class: Magento\Framework\Phrase\Renderer\MessageFormatter
translation result: Array
(
    [0] => There are {placeholder} messages requires your attention.
    [1] => There are {placeholder} messages requires your attention.
    [2] => There are {0} messages requires your attention.
)

render class: Magento\Framework\Phrase\Renderer\Placeholder
translation result: Array
(
    [0] => There are {placeholder} messages requires your attention.
    [1] => There are {placeholder} messages requires your attention.
    [2] => There are {0} messages requires your attention.
    [3] => There are {0} messages requires your attention.
)

render class: Magento\Framework\Phrase\Renderer\Inline
translation result: Array
(
    [0] => There are {placeholder} messages requires your attention.
    [1] => There are {placeholder} messages requires your attention.
    [2] => There are {0} messages requires your attention.
    [3] => There are {0} messages requires your attention.
    [4] => There are {0} messages requires your attention.
)

you can see that when it hits Magento\Framework\Phrase\Renderer\MessageFormatter it transforms to remove the labeled placeholder with a '0'.

The function that is doing this is the call to

$result = \MessageFormatter::formatMessage($this->translate->getLocale(), $text, $arguments);

in : Magento\Framework\Phrase\Renderer\MessageFormatter

this function comes from php intl, which on centos systems:

php -r "echo \MessageFormatter::formatMessage('en_US', 'There are {placeholder} messages requires your attention.', []);"

returns:

There are {0} messages requires your attention

and on ubuntu returns:

There are {placeholder} messages requires your attention

This is a pretty minor issue however in the vendor/magento/module-ui/view/base/web/js/grid/editing/editor.js:525

you have the following:

        countErrorsMessage: function () {
            return $t('There are {placeholder} messages requires your attention.')
                .replace('{placeholder}', this.countErrors());
        },

on centos systems that 'placeholder' won't exist once js translations are run because static content deploy usage of messageFormat has transformed it to a '0'

if you run php --ri intl on both systems you will see the libicu versions are different.

After some experimentation I found that libicu is not so simple to update for centos 7 in-fact its not a recommended thing to do outside of updating the OS to centos 8.

I am not sure if if the best solution is to update the message formatter to detect if the placeholder has been transformed and restore the named placeholders or maybe if we are translating with no arguments its not needed to run it through the messageFormat function at all.

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

[ ] Severity: S0 - Affects critical data or functionality and leaves users without workaround.
[ ] Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
[x] Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
[ ] Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
[ ] Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

m2-assistant[bot] commented 4 years ago

Hi @Luwdo. Thank you for your report. To help us process this issue please make sure that you provided the following information:

Summary of the issue
Information on your environment
Steps to reproduce
Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

Join Magento Community Engineering Slack and ask your questions in #github channel.

:warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

:clock10: You can find the schedule on the Magento Community Calendar page.

:telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

:movie_camera: You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

:pencil2: Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

orbitrod commented 4 years ago

@magento give me 2.4-develop instance

magento-engcom-team commented 4 years ago

Hi @orbitrod. Thank you for your request. I'm working on Magento 2.4-develop instance for you

magento-engcom-team commented 4 years ago

Hi @orbitrod, here is your Magento instance. Admin access: https://i-29268-2-4-develop.instances.magento-community.engineering/admin_ef59 Login: 23e98ea3 Password: 662abe20e391 Instance will be terminated in up to 3 hours.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 14 days if no further activity occurs. Is this issue still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? Thank you for your contributions!

mrtuvn commented 3 years ago

this is not stale

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 14 days if no further activity occurs. Is this issue still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? Thank you for your contributions!

magento / magento2