Update the CLDR sources

[daniellienert]

The CLDR data is used for example for formatting Dates, Currencies and for determining plural forms in various languages. The currently used Version 1.8.1 as of 2010-04-29 is pretty old.

While handling plurals and date formatting might not have changed much in the last 10 years. Some data as the country names have. The current version still contains East-Germany.

Some parts of the sources are pretty easy updatable, while others, like the plurals definitions completely changed its format.

• [x] Write functional test for relevant classes against the CLDR source (#2117)
• [x] Exchange the "main" section of the CLDR data (#2118)
• [ ] Exchange supplementary !!! At least the plurals format completely changed
• [ ] Exchange the rest of the files

[bwaidelich]

@daniellienert Thanks a lot for working on this, I added this ticket to the Neos 6.0 & Flow 7.0 Release Board. I would suggest to add a todo to update the documentation. Do you plan to continue working on this topic?

[sorenmalling]

ping @daniellienert :-)

Does the two todos in the issue here still remain? Do you need any help on any of those two topics?

[daniellienert]

@sorenmalling, yep the two issues still remain. I still plan to work on it. Help, well the plurals reader needs to be rewritten, or exchanged with a 3rd party (which I would recommend) eg: https://packagist.org/packages/wikimedia/cldr-plural-rule-parser. Can be done by anyone. @bwaidelich: Which documentation do you have in mind?

[sorenmalling]

Help, well the plurals reader needs to be rewritten, or exchanged with a 3rd party

I like that idea if the task rewriting the PuralReader is large (it's a coding and usage topic I haven't touched before).

I also found the repository here https://github.com/unicode-org/cldr/tree/master/common that could be worth mentioned and help us in future updates, to know where we get the files from

[bwaidelich]

Which documentation do you have in mind?

@daniellienert you wrote

!!! At least the plurals format completely changed

so I assume this will be a breaking change that should be documented in https://flowframework.readthedocs.io/en/stable/TheDefinitiveGuide/PartIII/Internationalization.html (and https://flowframework.readthedocs.io/en/stable/TheDefinitiveGuide/PartV/ReleaseNotes/index.html *g)

[daniellienert]

Ah, no that's a misunderstanding. The CLDR input file format completely changed. I haven't worked on it yet, but I am pretty confident, that we can exchange the file including its parser without changing any API or interfaces.

[bwaidelich]

Ah, thanks for clarifying, even better!

[bwaidelich]

@daniellienert FYI: since it's feature freeze time, today would be the last chance to get this one in

[bwaidelich]

@daniellienert Just checking: What's the status here? Can we mark this resolved and create follow-up issues for the remaining tasks or is this an all-or-nothing feature?

[bwaidelich]

@daniellienert ping

[bwaidelich]

Can anyone tell me whether it is OK to postpone the rest of the unfinished TODOs above in a different release?

[kdambekalns]

IMHO it should be "fine". It'll provide a mix of old and new data, but nothing is "broken in a breaking way…"

[albe]

What Karsten said. And I'm also in favor of using an existing cldr parser. It's one of those things we do because we had to, but it's really none of our "core business" of the Framework.

[bwaidelich]

OK, thanks for your feedback. I'll remove the issue from the release board for now but keep it open

[kdambekalns]

And I'm also in favor of using an existing cldr parser.

I never even looked elsewhere… 🤦‍♂️

https://packagist.org/packages/symfony/intl maybe?

[albe]

Yes, or the one Daniel mentioned from wikimedia: https://github.com/wikimedia/CLDRPluralRuleParser But let's keep that open for some other time whenever someone feels the itch and focus to dig into that.