Matching languages from osm-community-index to locales

[gravitystorm]

I'm going to spin this off from #3301. @tomhughes said:

Has anybody done a complete cross check between our language list and the OCI one?

So here we go. The list of languages in openstreetmap-website is driven by the list in translatewiki and OCI by the list in transifex, and unfortunately (but unsurprisingly) these don't match 1:1. There's some that are in both, some in one but not the other, and vice versa.

Lots of translations available in OCI are currently blank, i.e. have no translated strings. Those are marked with a * below.

A) Languages that are available in both openstreetmap-website and OCI

This covers most of the most popular languages, so that's good. Some are marked with a * since they contain no translations in OCI currently, but adding them to OCI will mean they will appear on openstreetmap-website with no issues.

• af ar ast az ba be bg bn br bs ca cs cy da de el en-GB en eo es et eu fa fi fr gl he hi hr hu ia id is it ja ka km kn ko lt lv mk mr ms my ne nl nn oc pa pl ps pt-BR pt ro ru sat sc sk sl sq sr sv ta te th tl tr tt uk vi yo* zh-CN zh-TW

B) Languages that are available in openstreetmap-website but not OCI

These are the ones that concern me most, because ideally we don't want our existing languages to be unsupported for the list of local chapters. Many of them are already requested in the openstreetmap id-editor transifex project but aren't currently active. There are, however, some edge cases among them.

• aln Gheg Albanian
• arz Egyptian Arabic
• be-Tarask Belarusian (Taraškievica orthography)
• ce Chechen
• diq Zazaki
• dsb Lower Sorbian
• fit Tornedalen Finnish
• fur Friulian)
• fy Western Frisian
• ga Irish
• gcf Guadeloupean Creole French
• gd Scottish Gaelic
• gsw Swiss German
• hsb Upper Sorbian
• kab Kabyle
• ksh Colognian
• ku-Latn Kurdish (Latin script)
• lb Luxembourgish
• mo Moldovan
• nb Norwegian Bokmål
• nds Low German
• nqo N’Ko
• pt-PT Portuguese Portgual
• scn Sicilian
• sco Scots
• sr-Latn Serbian (Latin script)
• xmf Mingrelian
• yi Yiddish
• zh-hk Chinese (Hong Kong)

C) Additional languages that are in OCI but aren't used by openstreetmap-website

For completeness, these are the other languages available in OCI, but many of these are blank. This list can help resolve some of the edge cases.

• ak am ar-AA as ay bg-BG bg@Cyrl bo ckb cv dv ee en-AU en-IN en-NZ fa-IR ff fil fr-FR gan grt gu ha he-IL hy ig jam jv kbd kha ki kk ks ku ky lg lij ln lo lus mg mi ml mn nd nl-NL no nr nso nv ny om or pap qu rm rw sd sg si sm sn so ss st su sw tg ti tk tn ts tum ug ur uz ve wo xh yue zh-HK zh zu*

So the main thing is to check if each of the languages listed in section B are genuinely not available in OCI, and if they are, resolve these edge cases where the language codes don't match. I'll post some of the edge cases that I've identified in the next comment.

[gravitystorm]

Some edge cases that I've spotted that could be resolved:

• We use zh-hk whereas OCI uses zh-HK. We need to either map between these on import, or fix one or the other if their capitalisation is incorrect
• We have a Moldovan translation in translatewiki, although Moldovan and Romanian are considered to be the same language. The mo code is not used in ISO-639-1, although it's reserved to avoid confusion. Perhaps we should remove mo from translatewiki completely.
• Translatewiki has pt and pt-br, but appears to export that to us as pt-PT and pt-BR. We have pt-BR, pt-PT and pt in our locales file, although the pt translation has not been updated for 8 years. I'm going to create a PR to remove the ancient pt translation, although this means that the mapping here (OCI has pt and pt-BR) will then break.

[tomhughes]

My answers on those:

• It should really be zh-HK because although matching is supposed to be case insensitive the documented canonical form is to uppercase the country.
• We probably need to ask Translatewiki why they still have mo as a separate language and then either remove it or rename it to ro-MD which would be the accurate name I guess.
• Arguably we should rename pt-PT to pt replacing the old one but there's no absolutely right answer on that.

[Firefishy]

en-GB should really be removed. Some translations also have incorrect old translations of the licensing text, specifically intro_3_1_html and credit_2_1_html. I asked awhile back: https://translatewiki.net/w/i.php?title=Support&offset=20220607083240#OpenStreetMap.org_Translation_Fixes_59953

[bgo-eiu]

pnb on Translate wiki (just committed today now that I've made it cross the threshold) is pa-PK in Transifex / OCI if that is helpful to know.