A tiny neural network for natural language detection.
This model is trained with data from the Tatoeba project, so it works well on short inputs too.
Architecture:
Details:
CLD3
ngrams are not hashed, as I think that kinda unpredictably confuses the model, as now different ngrams can have the same hash, so unimportant ngrams could be confused for important ones. Also that requires allocating a somewhat large key space for the hashes, while instead we want to be parsimonious with that, if we only need to remember 100 1-grams that's what we should be allocating memory for, but a small key space would get quickly crowded with colliding hashes, making it somewhat useless.CLD3
no embedding layer is used here, as I think it's not needed with this approach because the model is being given clearer signals, since each input always corresponds to a specific ngram.The following 50 languages are supported.
If you want to experiment you can tweak some internal constants and add more languages, remove languages (which necessarily improves accuracy for the remaining ones), tweak hyper-parameters for the neural network etc.
ISO 639-3 | Name |
---|---|
afr |
Afrikaans |
ara |
Arabic |
aze |
Azerbaijani |
bel |
Belarusian |
ben |
Bengali |
bul |
Bulgarian |
cat |
Catalan |
ces |
Czech |
ckb |
Kurdish |
cmn |
Mandarin |
dan |
Danish |
deu |
German |
ell |
Greek |
eng |
English |
est |
Estonian |
eus |
Basque |
fin |
Finnish |
fra |
French |
hau |
Hausa |
heb |
Hebrew |
hin |
Hindi |
hrv |
Croatian |
hun |
Hungarian |
hye |
Armenian |
ind |
Indonesian |
isl |
Icelandic |
ita |
Italian |
jpn |
Japanese |
kat |
Georgian |
kaz |
Kazakh |
kor |
Korean |
lit |
Lithuanian |
mar |
Marathi |
mkd |
Macedonian |
nld |
Dutch |
nob |
Norwegian |
pes |
Persian |
pol |
Polish |
por |
Portuguese |
ron |
Romanian |
run |
Rundi |
rus |
Russian |
slk |
Slovak |
spa |
Spanish |
srp |
Serbian |
swe |
Swedish |
tgl |
Tagalog |
tur |
Turkish |
ukr |
Ukrainian |
vie |
Vietnamese |
npm install --save lande
The library exports a single function which gives you a sorted list of detected languages and their probabilities.
import lande from 'lande';
lande ( 'What language is this sentence written in?' ); // => [['eng', 0.9999921321868896], ['deu', 0.000002357382982154377], ['heb', 0.000001461773877053929], ...]
lande ( 'In che lingua è scritta questa frase?' ); // => [['ita', 0.9999935626983643], ['ell', 0.0000025603442281862954], ['vie', 0.000002114558583343751], ...]
MIT © Fabio Spampinato