bittlingmayer
commented
4 years ago Open bittlingmayer opened 4 years ago
bittlingmayer
commented
4 years ago bittlingmayer
commented
4 years ago BERT_LANGS = [
'af', 'sq', 'ar', 'an', 'hy', 'ast', 'az', 'ba', 'eu', 'bar', 'be', 'bn', 'bpy', 'bs', 'br', 'bg', 'my', 'ca', 'ceb', 'ce', 'zh', 'zh-tw', 'cv', 'hr', 'cs', 'da', 'nl', 'en', 'et', 'fi', 'fr', 'gl', 'ka', 'de', 'el', 'gu', 'ht', 'he', 'hi', 'hu', 'is', 'io', 'id', 'ga', 'it', 'ja', 'jv', 'kn', 'kk', 'ky', 'ko', 'la', 'lv', 'lt', 'lmo', 'nds', 'lb', 'mk', 'mg', 'ms', 'ml', 'mn', 'mr', 'min', 'ne', 'new', 'nb', 'nn', 'oc', 'fa', 'pms', 'pl', 'pt', 'pa', 'ro', 'ru', 'sco', 'sr', 'sh', 'scn', 'sk', 'sl', 'azb', 'es', 'su', 'sw', 'sv', 'tl', 'tg', 'ta', 'tt', 'te', 'th', 'tr', 'uk', 'ur', 'uz', 'vi', 'vo', 'war', 'cy', 'fy', 'lah', 'yo'
]
| Wikipedia | Name | Notes |
|---|---|---|
| af | Afrikaans | |
| sq | Albanian | |
| ar | Arabic | |
| an | Aragonese | |
| hy | Armenian | |
| ast | Asturian | |
| az | Azerbaijani | |
| ba | Bashkir | |
| eu | Basque | |
| bar | Bavarian | |
| be | Belarusian | |
| bn | Bengali | |
| bpy | Bishnupriya Manipuri | |
| bs | Bosnian | |
| br | Breton | |
| bg | Bulgarian | |
| my | Burmese | |
| ca | Catalan | |
| ceb | Cebuano | |
| ce | Chechen | |
| zh | Chinese (Simplified) | zh-CN |
| zh-tw | Chinese (Traditional) | zh-HK, zh-MO |
| cv | Chuvash | |
| hr | Croatian | |
| cs | Czech | |
| da | Danish | |
| nl | Dutch | |
| en | English | |
| et | Estonian | |
| fi | Finnish | |
| fr | French | |
| gl | Galician | |
| ka | Georgian | |
| de | German | |
| el | Greek | |
| gu | Gujarati | |
| ht | Haitian | |
| he | Hebrew | Previously iw |
| hi | Hindi | |
| hu | Hungarian | |
| is | Icelandic | |
| io | Ido | |
| id | Indonesian | Previously in |
| ga | Irish | |
| it | Italian | |
| ja | Japanese | |
| jv | Javanese | Previously jw |
| kn | Kannada | |
| kk | Kazakh | |
| ky | Kirghiz | |
| ko | Korean | |
| la | Latin | |
| lv | Latvian | |
| lt | Lithuanian | |
| lmo | Lombard | |
| nds | Low Saxon | |
| lb | Luxembourgish | |
| mk | Macedonian | |
| mg | Malagasy | |
| ms | Malay | |
| ml | Malayalam | |
| mn | Mongolian | in Multilingual Cased (New) only |
| mr | Marathi | |
| min | Minangkabau | |
| ne | Nepali | |
| new | Newar | |
| nb | Norwegian (Bokmal) | Also no |
| nn | Norwegian (Nynorsk) | |
| oc | Occitan | |
| fa | Persian (Farsi) | |
| pms | Piedmontese | |
| pl | Polish | |
| pt | Portuguese | |
| pa | Punjabi | |
| ro | Romanian | |
| ru | Russian | |
| sco | Scots | |
| sr | Serbian | |
| sh | Serbo-Croatian | |
| scn | Sicilian | |
| sk | Slovak | |
| sl | Slovenian | |
| azb | South Azerbaijani | |
| es | Spanish | |
| su | Sundanese | |
| sw | Swahili | |
| sv | Swedish | |
| tl | Tagalog | Macrolanguage fil |
| tg | Tajik | |
| ta | Tamil | |
| tt | Tatar | |
| te | Telugu | |
| th | Thai | in Multilingual Cased (New) only |
| tr | Turkish | |
| uk | Ukrainian | |
| ur | Urdu | |
| uz | Uzbek | |
| vi | Vietnamese | |
| vo | Volapük | |
| war | Waray-Waray | |
| cy | Welsh | |
| fy | West Frisian | Macrolanguage fry |
| lah | Western Punjabi | Macrolanguage pan |
| yo | Yoruba |
Would be ideal to add the list of actual language codes to the multilingual README, so other systems can do lookups, and to prevent bugs where there is a discrepancy between the internal list and the documentation:
Instead of only: