Closed hugolpz closed 6 years ago
Github > NLP tool and data (Hermit Dave, 2016) is a tremendous source ! As the author attacked half a hundred language, "words" may linguistically unclean. Curration would be welcome.
IDEA : for each language, create a tinder like app, so native speaker may tag each word as real | artifact
. See https://github.com/wikimedia-france/Lingua-Libre/issues/14
French wordlist added. I used http://textmechanic.com/text-tools/basic-text-tools/remove-duplicate-lines/ to merge duplicate lemes.
cc: @tshrinivasan .
Note : Wikipedia being an encyclopedia is not the best corpus for educational vocabulary lists. Academics of the Subtlex made the argument that subtitles such as Open Subtitles are the most relevant for language learning as the stick the most to real oral speech.
Data can be downloaded...
...but appears to me as :
Central Institute for Indian Languages http://www.ciil.org/ http://www.ciil-lisindia.net/Tamil/Tamil_tech.html
See also :
NO DATA ONLINE ... #ridiculous #wasteTaxpayersMoney
Note:
1000 common words -- light cleanup to do - wordlist (text-file): words.txt
Tamil-English Basic Vocabulary, University of Pennsylvania -- double column table to clean up (bad list? seems like Thai, can someone with Tamil localization properly installed on their computer check?)
Contacted the author of Worldlex, asked them to free their data.
Existing words lists by frequency
Words lists: Various researches have produced high quality word frequency list. Most notables are :
Ongoing efforts
FRA-Lexique381-leme.txt,
FRA-Lexique381-ortho.txt
Indian languages
There should be some Indian language institutes providing lists for Indian languages.
Naming convention
See Wikipédia:Atelier identification/Nommage des photos d'animaux![screenshot from 2017-06-14 12-49-31](https://user-images.githubusercontent.com/1420189/27128723-04705286-5100-11e7-8077-e1720dc808a8.png)
Notes