Open Ali-H-Vahid opened 9 years ago
The europarl corpus was downloaded from http://www.statmt.org/europarl/v7/fr-en.tgz.
The English and French parts of the bilingual corpus were tokenised using the Moses Sample Tokeniser Version 1.1 by Pidong Wang (https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl) with the language parameters set to en
and fr
respectively and the flag -no-escape
activated.
The tokenised parts were truecased using the Moses Truecaser (https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/).
The results of this step can be found on demo-cngl:/home/scortes/projects/mm4w2v/data/europarl/*.true
.
1.5 hours were invested.
The C implementation of word2vec available on http://word2vec.googlecode.com/svn/trunk/ was used to compute the models. These were the parameters used:
-cbow 1
-size 800
-window 10
-negative 25
-hs 0
-sample 1e-4
-threads 16
-binary 1
-iter 15
-cbow 1
-size 200
-window 10
-negative 25
-hs 0
-sample 1e-4
-threads 16
-binary 1
-iter 15
The results of this step can be found on demo-cngl:/home/scortes/projects/mm4w2v/data/europarl/*.w2v_model.bin
.
The list of stop words was taken from the Terrier IR Platform version 4.0 (terrier-4.0/share/stopword-list.txt
). The script used to remove the stop words is in the repository, on https://github.com/Ali-H-Vahid/mm4w2v/blob/master/scripts/remove_stopwords.py. The option -p
that also removes non alphanumeric tokens was activated as well as the option -l
that lowercases both words and stop words before comparing them.
The results of this step can be found on demo-cngl:/home/scortes/projects/mm4w2v/data/europarl/*.without_stopwords
.
The results of this step can be found on demo-cngl:/home/scortes/projects/mm4w2v/data/europarl/*.first_5k
.
The results of this step can be found on demo-cngl:/home/scortes/projects/mm4w2v/data/europarl/*.bing_translated_to_fr
.