String import/export: cak / trs / zam / mix / meh is not a valid locale

[pocmo]

Output from Android2Po when importing/exporting strings:

    [failed] cak is not a valid locale
    [failed] trs is not a valid locale
    [failed] zam is not a valid locale

[pocmo]

• cak: Kaqchikel
  • http://www-01.sil.org/iso639-3/documentation.asp?id=cak
  • https://en.wikipedia.org/wiki/Kaqchikel_people
  • https://en.wikipedia.org/wiki/Kaqchikel_language
• trs: Chicahuaxtla Triqui
  • http://www-01.sil.org/iso639-3/documentation.asp?id=trs
  • https://es.wikipedia.org/wiki/Triqui_de_Chicahuaxtla
• zam: Miahuatlán Zapotec
  • http://www-01.sil.org/iso639-3/documentation.asp?id=zam
  • https://en.wikipedia.org/wiki/Miahuatl%C3%A1n_Zapotec

[pocmo]

Also see: https://bugzilla.mozilla.org/show_bug.cgi?id=1263437

[mastizada]

Babel do not support these locales, but I think we can do some custom tweaks for these locales in android2po (instead of using babel.Locale). Also adding custom locales for android doc should be checked (for values-xx directory). Ideally, these locales should be added to cldr unicode database, but that is different issue.

[pocmo]

That would be great. We have some additional Android runtime code to check whether a locale can be used (and to get the name of the language in other languages). We would need to check whether this works on any device with those locales.

[pocmo]

We have the same issue with the new locale "mix": Mixtepec Mixtec - http://www.mexico.sil.org/language_culture/mixtec/mixtec-mix.

[pocmo]

Android doesn't have a problem to build with cak / trs / zam / mix locales and use them at runtime. Of course it is unable to display the name of the languages. We would need to add those manually.

• @mastizada Do you have time to take care of patching android2po?
• @delphine Can you help us to get the name of the language in the language for cak / trs / zam / mix?

[mastizada]

Hi @pocmo , I think I will need a help this time :)

I already discussed that issue before, the Babel library do not support these locales and they are also absent in Unicode CLDR library (http://cldr.unicode.org/).

Looks like android2po uses this only for plural forms.

Currently there is an open issue in babel to support custom locales: https://github.com/python-babel/babel/issues/454 . They are planning since 2013 and the milestone for Babel 3 is not even started. The localedata.exists function looks for CLDR data files and returns UnknownLocaleError as there is no cat.dat file.

Possible solutions:

1. Adding dummy dat files to the babel directory (or better rewriting exists and load function to get files from project's or other directory) and having some basic rules for them.
2. Use some custom model for these locales instead of babel.Locale. That one looks a lot easier to do. For az locale there is only 1 plural rule (<PluralRule u'one: n in 1'> tested using a2po.py export az). Having languages similar to these locales in terms of plural rules can be helpful.

In both cases it will be just a monkeypatch to solve problem temporarily and should be removed once locale will be supported by babel or babel will support custom locales.

Which one should we test? :D

[pocmo]

I assume the babel directory for (1) is outside of the repository? If this means that every developer / build machine would have to do this then I definitely prefer (2). I haven't looked at the code of a2po too much -> Is the locale data used only for the plural form or are there other things that are needed?

In both cases it will be just a monkeypatch to solve problem temporarily and should be removed once locale will be supported by babel or babel will support custom locales.

Yeah, absolutely. :)

[mastizada]

Yes, they are under [venv or system path]/lib/python2.7/site-packages/babel/locale-data by default, but we can copy them to the project and use custom localedata.exists and loads functions to open them directly from that folder. Also we will need to prepare those files and documentation of CLDR is confusing.

There Language class in env.py file in a2po. It stores code and locale details, locale is an instance of babel.Locale. This locale variable used in env.py and commands.py and only locale.plural_form is used. So yes, although it has a lot of data, a2po only uses plural form rules and can be easily manipulated.

How we can learn plural rules for these locales? Also couldn't find similar locales in CLDR too, like other Mayan languages or languages used in Guatemala.

[Delphine]

@mastizada: plural rules for those locales can be found on Pontoon, top left of page

• https://pontoon.mozilla.org/cak/
• https://pontoon.mozilla.org/trs/
• https://pontoon.mozilla.org/zam/
• https://pontoon.mozilla.org/mix/ (let me know if you this to be more specific)

@pocmo: I'll get in touch with the localizers and get back to you about the name of the language in the language for cak / trs / zam / mix

[Delphine]

Seems like we now also have "meh" locale that will be affected by this. Plural rule: https://pontoon.mozilla.org/meh/ Locale name: Mixteco Yucuhiti

I'll also ask localizers how they say that locale name in their language, and will update here once I have that. thanks!

[Delphine]

Got an answer for a few of these locales, in regards to their names in their own languages:

• Triqui (trs): Triqui
• Mixtepec Mixtec (mix): Tu'un savi
• Maya Kaqchikel (cak): Kaqchikel I'm still waiting on Zapoteco (zam) and Mixtepec Mixtec (mix) at this point.

[Delphine]

• Mixteco Yucuhiti (meh): Tu´un savi ñuu Yasi'í Yuku Iti (!)

[Delphine]

Ok so I've got the answer for Zapoteco (zam), which was the last one we were waiting for. There are 2 possibilities:

• "International code": DíɁztè
• Name with Latin caracters: diiste The localizer tells me it would be great if it's possible to use the "international code" name.

[pocmo]

@Delphine Thanks! I'll add those to the language switcher.

@mastizada Do you have time to look at the import? Using a custom model instead of babel.Locale sounds like the best approach to me. It's also something we might be able to use easily for other langauges in the future.

[pocmo]

Thanks to @mastizada we can now import those strings.

@Delphine I see errors when importing mix and cak strings. For cak I filed a bug on bugzilla here: https://bugzilla.mozilla.org/show_bug.cgi?id=1406938 For mix I filed a bug in the "other" component. I could find one for "mix": https://bugzilla.mozilla.org/show_bug.cgi?id=1406940

[mastizada]

@pocmo should we check for html tags while verifying po files (check_translations)?

[pocmo]

@mastizada Might make sense yeah. As long as we are importing manually the errors are quite visible though. Let's file a separate issue for that. :)

[pocmo]

Issues fixed. Moved to Granite.

[pocmo]

Follow-up issue to track the translation errors: #1456