search

facebookresearch / LASER

Language-Agnostic SEntence Representations
Other
3.6k stars 463 forks source link

Clarification for Chinese language variants #225

Closed raunaksinhacisco closed 1 year ago

raunaksinhacisco commented 1 year ago

There are three variants of Chinese that are listed a supported languages in LASER -

  1. Chinese (zh)
  2. Yue Chinese (yue)
  3. Wu Chinese (Wu)

In platforms like Google usually Chinese is available in three forms -

  1. Chinese (Simplified, China, zh-CN)
  2. Chinese (Traditional, Taiwan, zh-TW)
  3. Chinese (Traditional, Hong Kong, zh-HK)

Can someone please help me with the following questions -

  1. Is Chinese (zh) used for both traditional and simplified Chinese?
  2. Yue and Wu Chinese are verbal dialects the closest written form of Yue is Cantonese (denoted by Chinese (Traditional, Hong Kong, zh-HK?) . How are non-written dialects used in LASER?
heffernankevin commented 1 year ago

Hi @raunaksinhacisco! The version of Chinese would depend on the training sets used, and unfortunately indeed the .zh language code is underspecified. However, the more recent LASER3 encoders do have explicit support for both Simplified and Traditional Chinese! In order to embed using these specific models you can perform the following:

  1. Go to the LASER3 (NLLB) model page and download the Simplified and Traditional Chinese models. This is done using the following command: bash ./download_models.sh zho_Hans zho_Hant. NOTE: "zho_Hans" and "zho_Hant" are the FLORES200 language codes for simplified and traditional Chinese respectively.

Regarding Yue and Wu Chinese training sets, there are bitexts available from sources such as: https://opus.nlpl.eu/Tatoeba.php

Hope this helps!