Disable custom fonts for unsupported locales

[zaach]

Rather than disable them across the board (#1128), let's disable them for locales with unsupported character sets.

[johngruen]

new fira: http://dev.carrois.com/en/fira-3-1/

[pdehaan]

New Fira 3.1 tracking bug: #1199

[zaach]

Here's a list of supported languages I gleaned from the technical report:

var langs = [
"Afar"
"Afrikaans"
"Albaian"
"Asu"
"Basque"
"Bemba"
"Bena"
"Bosnian"
"Breton"
"Catalan"
"Chiga"
"Cornish"
"Crimean Tatar"
"Croatian"
"Czech"
"Danish"
"Dutch"
"Embu"
"English"
"Esperanto"
"Estonian"
"Faroese"
"Filipino"
"Finnish"
"French"
"Frisian West"
"Friulian"
"Ga"
"Galician"
"Ganda"
"German"
"Gusii"
"Hungarian"
"Icelandic"
"Indonesian"
"Interlingua"
"Irish"
"Italian"
"Jju"
"Kabucerdianu"
"Kalaallisut"
"Kalenjin"
"Kamba"
"Kikuyu"
"Kinyarwanda"
"Koro"
"Kurdish"
"Latvian"
"Lithuanian"
"Low German"
"Luo"
"Luyia"
"Machame"
"Makonde"
"Malagasy"
"Malay"
"Maltese"
"Manx"
"Maori"
"Meru"
"Moldavian"
"Morisyen"
"NorthNdebele"
"Norwegian Bokål"
"Norwegian Nynorsk"
"Nyanja"
"Nyankole"
"Occitan"
"Oromo"
"Polish"
"Portuguese"
"Romanian"
"Romansh"
"Rombo"
"Rwa"
"Saho"
"Samburu"
"Sami Northern"
"Sango Sena"
"Serbian"
"Shambala"
"Shona"
"Sidamo"
"Slovak"
"Slovenian"
"Soga"
"Somali"
"Sorbian,Lower"
"Sorbian,Upper"
"Sotho,Northern"
"Sotho,Southern"
"South Ndebele"
"Spanish"
"Swahili"
"Swati"
"Swedish"
"Swiss German"
"Taita"
"Taroko"
"Teso"
"Tsonga"
"Tswana"
"Turkish"
"Tyap"
"Vunjo"
"Welsh"
"Wolof"
"Xhosa"
"Zulu"
"Azerbaijani"
"Bashkir"
"Belarusian"
"Bulgarian"
"Buriat"
"Chechen"
"Kazakh"
"Macedonian"
"Mongolian"
"Russian"
"Serbian"
"Tajik"
"Tatar"
"Uighur"
"Ukrainian"
"Uzbek"
"Greek"
];

Next step is to compare this list to the locales we support.

[shane-tomlinson]

I have opened PR #1222 to update the fonts to fira v3.1. Is there a list of locales where we support the language but the font does not support the character set?

I am using the list of supported languages from the technical pack for FiraSans-Light available at http://www.carrois.com/wordpress/downloads/fira_3_1/FiraPDF3108.zip and cross referencing that list with http://www.unicode.org/cldr/charts/25/summary/root.html

• an - aragonese
• ar - arabic
• as - Assamese
• ast - Asturian
• bn-BD - Bengali
• bn-IN - Bengali
• fa - Persian

(note, this is in progress, I have checked up to fa

Note, maybe even better is http://www.unicode.org/cldr/charts/25/supplemental/locale_coverage.html

[shane-tomlinson]

Our full list of supported locales:

{ af: 'Afrikaans',
  an: 'Aragonese',
  ar: 'Arabic',
  as: 'Assamese',
  ast: 'Asturian',
  be: 'Belarusian',
  bg: 'Bulgarian',
  bn_bd: 'Bengali',
  bn_in: 'Bengali',
  br: 'Breton',
  bs: 'Bosnian',
  ca: 'Catalan',
  cs: 'Czech',
  cy: 'Welsh',
  da: 'Danish',
  de: 'German',
  el: 'Greek',
  en_gb: 'U.K. English',
  en_us: 'U.S. English',
  en_za: 'English',
  eo: 'Esperanto',
  es: 'Spanish',
  es_ar: 'Spanish',
  es_cl: 'Spanish',
  es_mx: 'Mexican Spanish',
  et: 'Estonian',
  eu: 'Basque',
  fa: 'Persian',
  ff: 'Fulah',
  fi: 'Finnish',
  fr: 'French',
  fy: 'Western Frisian',
  fy_nl: 'Western Frisian',
  ga: 'Irish',
  ga_ie: 'Irish',
  gd: 'Scottish Gaelic',
  gl: 'Galician',
  gu: 'Gujarati',
  gu_in: 'Gujarati',
  he: 'Hebrew',
  hi_in: 'Hindi',
  hr: 'Croatian',
  ht: 'Haitian',
  hu: 'Hungarian',
  hy_am: 'Armenian',
  id: 'Indonesian',
  is: 'Icelandic',
  it: 'Italian',
  it_ch: 'Italian',
  ja: 'Japanese',
  kk: 'Kazakh',
  km: 'Khmer',
  kn: 'Kannada',
  ko: 'Korean',
  ku: 'Kurdish',
  lij: 'Ligurian',
  lt: 'Lithuanian',
  lv: 'Latvian',
  mai: 'Maithili',
  mk: 'Macedonian',
  ml: 'Malayalam',
  mr: 'Marathi',
  ms: 'Malay',
  nb_no: 'Norwegian Bokmål',
  ne_np: 'Nepali',
  nl: 'Dutch',
  nn_no: 'Norwegian Nynorsk',
  or: 'Oriya',
  pa: 'Punjabi',
  pa_in: 'Punjabi',
  pl: 'Polish',
  pt: 'Portuguese',
  pt_br: 'Brazilian Portuguese',
  pt_pt: 'European Portuguese',
  rm: 'Romansh',
  ro: 'Romanian',
  ru: 'Russian',
  si: 'Sinhala',
  sk: 'Slovak',
  sl: 'Slovenian',
  son: undefined,
  sq: 'Albanian',
  sr: 'Serbian',
  sr_latn: 'Serbian',
  sv: 'Swedish',
  sv_se: 'Swedish',
  ta: 'Tamil',
  te: 'Telugu',
  th: 'Thai',
  tr: 'Turkish',
  uk: 'Ukrainian',
  ur: 'Urdu',
  vi: 'Vietnamese',
  xh: 'Xhosa',
  zh_cn: 'Chinese',
  zh_tw: 'Chinese',
  zu: 'Zulu' }

To get this, I used the data available on http://unicode.org/repos/cldr/trunk/common/main/en.xml to map locale->english name. If the locale contains a region that was not found in the XML list, I used only the language.

Now, time to compare this to Zach's list. Computers are awesome.

[pdehaan]

Computers? Gah! I've been printing off all this stuff and been reviewing it by hand all this time!

[shane-tomlinson]

Updated list of unsupported locales:

unsupported locales:
 * an (aragonese)
 * ar (arabic)
 * as (assamese)
 * ast (asturian)
 * bn_bd (bengali: bangladesh)
 * bn_in (bengali: india)
 * fa (persian)
 * ff (fulah)
 * gd (scottish gaelic)
 * gu (gujarati)
 * gu_in (gujarati: india)
 * he (hebrew)
 * hi_in (hindi: india)
 * ht (haitian)
 * hy_am (armenian: armenia)
 * ja (japanese)
 * km (khmer)
 * kn (kannada)
 * ko (korean)
 * lij (ligurian)
 * mai (maithili)
 * ml (malayalam)
 * mr (marathi)
 * ne_np (nepali: nepal)
 * or (oriya)
 * pa (punjabi)
 * pa_in (punjabi: india)
 * si (sinhala)
 * son (unknown)
 * ta (tamil)
 * te (telugu)
 * th (thai)
 * ur (urdu)
 * vi (vietnamese)
 * zh_cn (chinese: china)
 * zh_tw (chinese: taiwan)

[shane-tomlinson]

A blacklist:

[ 'an',
  'ar',
  'as',
  'ast',
  'bn_bd',
  'bn_in',
  'fa',
  'ff',
  'gd',
  'gu',
  'gu_in',
  'he',
  'hi_in',
  'ht',
  'hy_am',
  'ja',
  'km',
  'kn',
  'ko',
  'lij',
  'mai',
  'ml',
  'mr',
  'ne_np',
  'or',
  'pa',
  'pa_in',
  'si',
  'son',
  'ta',
  'te',
  'th',
  'ur',
  'vi',
  'zh_cn',
  'zh_tw' ]

Or, would it be better to do a whitelist here?

[shane-tomlinson]

BTW, my code to calculate the list is in https://github.com/shane-tomlinson/fxa-content-server-supproted-languages

[johngruen]

@shane-tomlinson blacklisting seems like the way to go since we can track a shrinking rather than a growing list.

[ckarlof]

To discuss out our FxA Web standup this morning.

[shane-tomlinson]

@zaach - are you taking this or do you want me to?

[ckarlof]

@shane-tomlinson suggests that #1266 could be fixed in conjunction with this (don't use web fonts if we're in IE8).