search

iLanguage / ilanguagelab

Automatically exported from code.google.com/p/ilanguagelab
0 stars 0 forks source link

Extract the vocabulary from the Inuktitut Corpus #14

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
Purpose of addition of this task:
We need a list of (probable words) for any task dealing with Inuktitut. To get 
the list we plan on using Inuktitut Magazine (since it is representative 
literature). We would like to extract all the words from the magazine, along 
with their frequency information which can give a clue as to their function vs 
content word status, and their likelyhood when spell checking a document or 
doing any predictive task. This word list will be further processed to find 
roots for the Inuktitut Search, and to find morphemes for the Inuktitut spell 
checker.

After the review, please add a Inuktitut Magazine section to Inuktitut Corpus 
wiki page.
*detailing the number of words per language in the corpus
*detailing the average number of words per file
*maybe some statistics about how dense the vocabulary is per file (ie dense or 
diverse vocab)
*Discussing the quality and drawbacks of the corpus

After the review, the expected next step is to:
*Generate jape grammars to highlight words in context 
*Work on morphological parsing of the vocab

Original issue reported on code.google.com by a...@ilanguage.ca on 9 Nov 2011 at 10:34

GoogleCodeExporter commented 9 years ago

Original comment by a...@ilanguage.ca on 10 Nov 2011 at 12:32

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
The wordlist from InuktitutMagazine vol 77-104 
https://github.com/cesine/CorpusInuktitut/blob/master/gen/InuktitutMagzine_word_
frequencies_matrix_by_document.csv

The script that was used to create the matrix:
http://code.google.com/p/ilanguagelab/source/browse/src/com/fieldlinguist/groovy
/ExtractTermFrequenciesMatrix.groovy

The script that was used to create the word frequency files
http://code.google.com/p/ilanguagelab/source/browse/src/com/fieldlinguist/groovy
/ExtractWordsOrderBySuffix.groovy

Original comment by gina.c.c...@gmail.com on 25 Nov 2011 at 4:48

GoogleCodeExporter commented 9 years ago

Original comment by a...@ilanguage.ca on 25 Nov 2011 at 10:25