Language identification - 'sh' code

[einrogerst]

The fastText language identification models support language code 'sh' (https://fasttext.cc/docs/en/language-identification.html). However, this code is not listed in the ISO codes (https://www.loc.gov/standards/iso639-2/php/code_list.php). It is unclear if it refers to Shan language (shn), Shona language (sna), or any other language.

[rspeer]

I'm not a fasttext developer, but I came across this.

• Don't use ISO 639. It's very old and it's not a complete enough standard for language codes on the modern Internet. The IETF standard is named BCP 47.
• You can look up BCP 47 language tags with the langcodes Python library or on a web page like this: https://r12a.github.io/app-subtags/

sh is the code for Serbo-Croatian. It's vaguely deprecated and is considered equivalent to Serbian (sr), Croatian (hr), or Bosnian (bs), three highly-related but politically-distinct languages that are mostly indistinguishable in text.