search

mozilla-mobile / fenix

⚠️ Fenix (Firefox for Android) moved to a new repository. It is now developed and maintained as part of: https://github.com/mozilla-mobile/firefox-android
https://github.com/mozilla-mobile/firefox-android
Mozilla Public License 2.0
6.47k stars 1.27k forks source link

Before Release: add Release-only locales to list #7104

Closed liuche closed 4 years ago

liuche commented 4 years ago

Why/User Benefit/User Problem

Currently, we will bundle the same locales in Nightly and Preview, regardless of what status of translation they are in.

However, before we make our main release, we need to make sure only release-approved locales are included. They should be on Nightly and Beta, but not Production builds.

Acceptance Criteria (how do I know when I’m done?)

Before July 18:

On July 20:

L10N strings uplift discussion L10N locale doc

┆Issue is synchronized with this Jira Task

liuche commented 4 years ago

@vesta0 will we want to filter out partially-translated locales after Beta Migration, or do we want to keep all the translated strings? Filtering will mean that users in that locale will see English-only, rather than a (low percentage) partial translation. We'll have to contact @Pike soon if we want to get that list.

vesta0 commented 4 years ago

@liuche do we how what percentage of strings have been translated? How far are we from Fennec parity in term of localization?

liuche commented 4 years ago

Here's the current progress on Pontoon (and I think Fenix is the main Android project): https://pontoon.mozilla.org/projects/android-l10n/

What do you think, @Pike (or Jeff Beatty but I don't know his github) - do we know enough about the localization languages progress compared to Fennec to make this release list?

Pike commented 4 years ago

I'll punt this to @gueroJeff

gueroJeff commented 4 years ago

Here's the current progress on Pontoon (and I think Fenix is the main Android project): https://pontoon.mozilla.org/projects/android-l10n/

This is the link I've been using: https://pontoon.mozilla.org/projects/android-l10n/tags/fenix/ What do you think, @Pike (or Jeff Beatty but I don't know his github) - do we know enough about the localization languages progress compared to Fennec to make this release list?

We're not going to reach parity, just by virtue of some locales shipping on Fennec chose not to participate in localizing Fenix. The metric we're using is average completion across all locales. Fennec was at 90% across all locales, Fenix is currently at 75% average.

To answer the earlier question, all products ship partial localizations. That's part of the risk that accompanies community localization. Fenix should follow that practice too.

liuche commented 4 years ago

@gueroJeff will review the locales and decide on what the cutoff for inclusion with the Release build. Once he gives us the criteria (and the locales?) we'll be able to update the script to include the release locales.

gueroJeff commented 4 years ago

I'm going to set the cutoff at 70% complete. There's some automation the Bedrock team implemented to determine how complete a localization is and then automatically push it to prod when it reaches a certain threshold. Similar automation here would be preferred, but if we're unable to get that for the launch, I can produce a manual list at l10n freeze.

liuche commented 4 years ago

We'll need to uplift the new localizations up until July 18, which will be after code freeze (which is okay because it's new strings only). After we know the localization rate, we'll be able to determine the locales to include/filter.

However, right now, we can still figure out how to make this work for the current set of locales, and just update the specific set of locales once we know their completion rates.

person808 commented 4 years ago

I downloaded an apk from here since it looks like we run a script in taskcluster to remove locales that aren't in the locale list. Then I manually went through the languages and checked off the locales I saw in the l10n config. Some locales were missing, maybe related to #12613?

The missing locales are kn, ms, ne-NP, nv, uz. Of these only kn appears to have more than 70% completion. ms, ne-NP, and nv have 0% completion so they are probably missing because the translation strings don't even exist. uz has 13% completion so I'm not sure why it doesn't show up (although we should remove it from the locale list anyway)

Delphine commented 4 years ago

I currently don't see either Uzbek (uz) nor Kannada (kn) on the latest Nightly. They both have partial localizations, so this is maybe also related to #12613?

person808 commented 4 years ago

We don't have the strings for Uzbek or Kannada in the app yet. Looks like they're just missing due to lag in getting the translations pulled in. The TC script to strip locales works on my machine so it looks like the first part of this issue is verified.

liuche commented 4 years ago

@Delphine will you comment with the list of locales that we need to include, once you have them? Or should we go through the Pontoon list on Monday when we make the final list change?

Delphine commented 4 years ago

@liuche Yes I can comment here once I have the definite list. Monday early morning (PST) still works for you?

Delphine commented 4 years ago

Out of the 90 locales on Pontoon, only 5 have not made the 70% threshold for making it into release: Bulgarian (bg), Uzbec (uz), Navajo(nv), Malay (ms) and Nepali (ne-NP).

The list of 85 locales needed for release are therefore: "an", "ar", "ast", "az", "be", "bn", "br", "bs", "ca", "cak", "co", "cs", "cy", "da", "de", "dsb", "el", "en-CA", "en-GB", "eo", "es", "es-AR", "es-CL", "es-ES", "es-MX", "et", "eu", "fa", "ff", "fi", "fr", "fy-NL", "ga-IE", "gd", "gn", "gu-IN", "he", "hi-IN", "hr", "hsb", "hu", "hy-AM", "id", "is", "it", "ja", "ka", "kab", "kk", "kn", "ko", "lij", "lo", "lt", "ml", "mr", "my", "nb-NO", "nl", "nn-NO", "oc", "pa-IN", "pl", "pt-BR", "pt-PT", "rm", "ro", "ru", "sk", "sl", "sq", "sr", "su", "sv-SE", "ta", "te", "th", "tr", "trs", "uk", "ur", "vec", "vi", "zh-CN", "zh-TW"

Please note that we've ran into an issue with some a-c strings for addons not making it into l10n, which means some menu items are currently untranslated: https://github.com/mozilla-mobile/android-components/issues/7777

liuche commented 4 years ago

This will be ready for QA once we cut the Release Candidate for the Firefox Production Migration. We will want to verify that in the language picker, only the approved 70%+ locales are included (see delphine's comment above for that list).

sflorean commented 4 years ago

@liuche started testing on 79.0.0 AC 48.0.8 RC2 and a couple of locales are missing, here are some of them: an, bn, bs, cs, dsb, el, eo, et, fa, ff, gd, gn, he, hsb, hy-Am, id, is, kk, kn, lij, lt, ml, mr, my. On what RC should we test this?

Delphine commented 4 years ago

So I've tested on this 79 build (that I got from https://firefox-ci-tc.services.mozilla.com/tasks/index/mobile.v2.fenix.fennec-production.2020.07.24.revision), and I'm seeing a few locales missing (not all the ones mentioned above): bn, dsb, he, hsb, id Has there been an issue cutting off locales for the release?

person808 commented 4 years ago

I see dsb and hsb as Lower Sorbian and Upper Sorbian. bn is Bangla I think? I don't see he or id in that apk

liuche commented 4 years ago

@person808 any ideas on what might be happening here? are these getting filtered out by accident, or are they not being included/imported at all?

person808 commented 4 years ago

Not sure. When I run the filtering script locally on the releases/v79.0.0 branch, the only locales that get removed are bg, in, iw, and uz. Taskcluster should run the scripts for fennec-beta and fennec-production (see kind.yml and look for filter-incomplete-translations)

Edit: Wait the values-id and values-he resource folders appear to be missing

flodolo commented 4 years ago

Not that I know a lot about Android, but https://developer.android.com/reference/java/util/Locale.html

Locale's constructor has always converted three language codes to their earlier, obsoleted forms: he maps to iw, yi maps to ji, and id maps to in. This continues to be the case, in order to not break backwards compatibility.

Pike commented 4 years ago

Those are mapped in https://github.com/Pike/compare-locales/blob/e3eab8131efa2d2a2c7c81827b8ff9d7ac2f2d9a/compare_locales/paths/matcher.py#L13-L19 between project config and disk paths. If either he or id are not shipping, you need to prune the corresponding values-iw or values-in folder.

sflorean commented 4 years ago

Testing results: "an", Aragonese - displayed "ar", Arabic - displayed "ast", Asturian - displayed "az", Azerbaijani - displayed "be", Belarusian - displayed "bn", Bangla - displayed "br", Breton - displayed "bs", Bosnian - displayed "ca", Catalan - displayed "cak", Cak - displayed "co", Corsican - displayed "cs", Cestina - displayed "cy", Cymraeg - displayed "da", Dansk - displayed "de", Deutsch - displayed "dsb", displayed "el", Greek - displayed "en-CA", - displayed "en-GB", - displayed "eo", Esperanto - displayed "es", Espanol - displayed "es-AR", Espaniol Argentina - diplayed "es-CL", Espaniol Chile - displayed "es-ES", Espaniol Espana - displayed "es-MX", Espaniol Mexico - displayed "et", Eesti - displayed "eu", Basque/Euskara - displayed "fa", Persian - displayed "ff", Pullaar (fulah) - displayed "fi", Suomi - displayed "fr", Francais - displayed "fy-NL", West-Frysk - displayed "ga-IE", Gaeilge - displayed "gd", Scottish Gaelic- displayed "gn", Guarani - displayed "gu-IN", Gujarati India - displayed "he", Hebrew? not displayed "hi-IN", Hindi India - displayed "hr", Hrvatski - displayed "hsb", displayed "hu", Magyar - displayed "hy-AM", Armenian - displayed "id", Indonesian? not displayed "is", Islenska - displayed "it", Italian - displayed "ja", Japanese - displayed "ka", Georgian - displayed "kab", Kabyle - displayed "kk", Kazakh - displayed "kn", Kannada - displayed "ko", Korean - displayed "lij", Ligurian - displayed "lo", Lao - displayed "lt", Lithuanian - displayed "ml", Malayalam - displayed "mr", Marathi - displayed "my", Burmese - displayed "nb-NO", Norsk bokmal - displayed "nl", Nederlands - displayed "nn-NO", Nynorsk - displayed "oc", Occitan - displayed "pa-IN", Punjabi India - displayed "pl", Polski - displayed "pt-BR", Portugues Brazil - displayed "pt-PT", Portugues Portugal - displayed "rm", Rumantsch - displayed "ro", Romanian - displayed "ru", Russian - displayed "sk", Slovak - displayed "sl", Slovenian - displayed "sq", Shqip - displayed "sr", Serbian - displayed "su", Sundanase - displayed "sv-SE", Svenska Sverige - displayed "ta", Tamil - displayed "te", Telugu - displayed "th", Thai - displayed "tr", Turkish - displayed "trs", Trs - displayed "uk", Ukrainian - displayed "ur", Urdu - displayed "vec", Venetian displayed "vi", Vietnamese - displayed "zh-CN", Chinese China - displayed "zh-TW" Chinese Taiwan - displayed

Note:

@liuche @Delphine

gueroJeff commented 4 years ago

There are also inconsistencies in that list. Several display names are the locale code, not the language name. Several are not in English (e.g., "Polski") and the two Chinese should display as Simplified Chinese (zh-CN) and Traditional Chinese (zh-TW).

Pike commented 4 years ago

We don't have control over the natural names, those are coming from Android (and I think that's a feature).

For Hebrew and Indonesian, we should translate the codes when reading the release locales, probably. I'd not make the config list be different. Filed #13066 on that.

person808 commented 4 years ago

Ah I see. Looks like l10n.toml contains "he" and "id" instead of "iw" and "in" so the folders are being deleted. Will file PRs to update the script so its aware of the mappings in the release branch and master

person808 commented 4 years ago

I'll close this since the filtering appears to work except for Hebrew and Indonesian. We can verify that they are present when we QA #13066