Closed aflueckiger closed 3 years ago
Models
/mnt/storage/clwork/projects/climpresso/local_storages/harlie/federal-gazette/embedding_vXXX
python2
fpath = '/mnt/storage/clwork/projects/climpresso/local_storages/harlie/federal-gazette/embedding_v3/biskip.300.de-fr.bin'
from multivec import MonolingualModel, BilingualModel
model = BilingualModel(fpath)
model.trg_model
model.trg_model.word_vec('france')
# compare cross-lingual
model.trg_closest('bündnerfleisch')
model.src_closest('ordinateur')
The embeddings of some words with old spelling seem to be very noisy. Moreover, they are oddly similar to each other without having an apparent relation.
In [35]: model.trg_closest('abtheilung')
Out[35]:
[('m\xc3\xb4me', 0.7755178213119507),
('clans', 0.769781231880188),
('do', 0.763927161693573),
('ot', 0.7473583817481995),
('section', 0.7450423836708069),
('celte', 0.7448492646217346),
('dn', 0.7437467575073242),
('ensorte', 0.7386529445648193),
('lo', 0.7359005212783813),
('sections', 0.7342149615287781)]
In [36]: model.trg_closest('bundesrath')
Out[36]:
[('m\xc3\xb4me', 0.8397364616394043),
('celte', 0.8255298137664795),
('mr', 0.8033562898635864),
('\xe2\x80\x9e', 0.7990537285804749),
('do', 0.7959240674972534),
('qne', 0.7946202754974365),
('clans', 0.7938278913497925),
('sou', 0.7881290912628174),
('d\xc3\xa8s-lors', 0.7880147695541382),
('lo', 0.7871352434158325)]
In [37]: model.trg_closest('theilnahme')
Out[37]:
[('m\xc3\xb4me', 0.7302607297897339),
('celte', 0.7289631366729736),
('clans', 0.7195239067077637),
('non-seulement', 0.7147542238235474),
('daus', 0.7087118029594421),
('\xe2\x80\x9e', 0.7065469622612),
('uu', 0.7032425403594971),
('d\xc3\xa8s-lors', 0.7022230625152588),
('sou', 0.6991384029388428),
('qne', 0.6973485350608826)]
Maybe the older texts are not large enough?
Or should we tryout https://github.com/facebookresearch/MUSE
@simon-clematide
They are definitely not OOV since I can look up them. All models are lowercase by the way. I am investigating this issue. In general, the cross-lingual embeddings work nicely.
I will also have a look at MUSE, yet I am rather skeptical that they work better. So far we leverage the parallelism of the corpus instead of simply performing a matrix transformation with Procrustes.
EDIT: may be not enough material as you mention. However, I am also skeptical about this.
This is an evaluation of the best cross-lingual alignments with the supervised MUSE library based on kNN with cosine similarity (corresponding CSLS value, maybe slightly higher in other iteration). The evaluation is performed on the bilingual dictionary provided in MUSE (out-of-domain).
Please note that the comparison between FastText and MultiVec is not meaningful currently due to different minimal occurrences, thus other coverage of validation. Moreover, the randomization of the corpus is not the same for all experiments
FastText: -thread 20 -minCount 5 -ws 10 -dim 300 -minn 3 -maxn 6 -epoch 50 MultiVec: --window-size 10 --sg
Model | Dim | Train Params | Eval Params | Aligning | NN Precision @ 1 | NN Precision @ 5 | CSLS Precision @ 1 | CSLS Precision @ 5 |
---|---|---|---|---|---|---|---|---|
FastText_v1 | 300 | no subword, iter 10, min-count 5 | max vocab 200k | MUSE | 51.97 | 67.78 | 61.97 | 74.70 |
FastText_v2 | 300 | subword 3-6, iter 50, min-count 5 | max vocab 200k | MUSE | 54.06 | 68.65 | 61.58 | 74.38 |
FastText_v2 | 300 | subword 3-6, iter 50, min-count 5 | max vocab 200k | RCSLS | 56.36 | 71.73 | 62.94 | 74.47 |
multivec_v4 | 300 | iter 20, min-count 10 | max vocab 200k | parallelism + MUSE | 68.22 | 78.29 | 73.02 | 83.88 |
multivec_v4 | 300 | iter 20, min-count 10 | max vocab 200k | parallelism | 69.30 | 78.76 | 73.49 | 84.81 |
multivec_v5 | 300 | iter 50, min-count 10 | max vocab 200k | parallelism | 69.66 | 78.33 | 73.37 | 84.52 |
multivec_v6 | 300 | iter 50, min-count 5 | max vocab 200k | parallelism | 65.07 | 78.21 | 71.64 | 85.08 |
multivec_v6 | 300 | iter 50, min-count 5 | entire vocab | parallelism | 67.48 | 77.93 | 74.0 | 86.30 |
multivec_v7 | 150 | iter 20, min-count 5 | max vocab 200k | parallelism | 61.79 | 74.63 | 65.67 | 80.23 |
--- | --- | --- | --- | --- | --- | --- | --- | --- |
multivec_new_corp_v1 | 100 | iter 20, min-count 5 | entire vocab | parallelism | 66.32 | 78.06 | 68.22 | 80.48 |
multivec_new_corp_v1 | 150 | iter 20, min-count 5 | entire vocab | parallelism | 69.95 | 82.21 | 72.62 | 85.31 |
multivec_new_corp_v1 | 300 | iter 20, min-count 5 | entire vocab | parallelism | 69.43 | 79.70 | 75.22 | 86.70 |
Model | Dim | Train Params | Eval Params | Aligning | NN Precision @ 1 | NN Precision @ 5 | CSLS Precision @ 1 | CSLS Precision @ 5 |
---|---|---|---|---|---|---|---|---|
multivec_v6 | 300 | iter 50, min-count 5 | entire vocab | parallelism | 43.02 | 59.88 | 39.53 | 56.98 |
--- | --- | --- | --- | --- | --- | --- | --- | --- |
multivec_new_corp_v1 | 100 | iter 20, min-count 5 | entire vocab | parallelism | 38.06 | 58.06 | 36.12 | 52.90 |
multivec_new_corp_v1 | 150 | iter 20, min-count 5 | entire vocab | parallelism | 42.58 | 65.16 | 41.29 | 65.80 |
multivec_new_corp_v1 | 300 | iter 20, min-count 5 | entire vocab | parallelism | 49.03 | 66.45** | 47.10 | 68.39 |
The bilingual in-domain dictionary was created by sampling from the Moses Translation file with the following command.
more mt_moses/train_de_fr/model/lex.e2f | awk '{print $3" "$2" "$1}' | uniq -f2 | egrep "0\.0{1,2}[1-9][0-9]+ [a-z]{7,15} [a-z]{7,15}" | sort -r | more | shuf -n 10000 | awk '{print $3" "$2}' | egrep -v "-" > ../../dico-de-fr-sample.txt
To provide high-quality translations the sampled items where manually filtered and extended with some prominent translations. The final dataset consists of 191 pairs of words (DE-FR).
We have trained various models with different hyperparameters, pre-processing, and data. Below there is an evaluation of the jointly trained German Embeddings. The evaluation is performed on lowercased analogies of a generic dataset (i.e., out of domain evaluation).
python3 lib/eval_embeddings.py $FILE.VEC --lowercase
embedding_v1 data: only FedGaz preprocessing: --lowercase --normalize-digits --normalize-punk --min-count 10 dimension: 100/300 window size: 5 min count: 10 iterations: 10
embedding_v2 data: FedGaz + Europarl preprocessing: --lowercase --normalize-digits --normalize-punk --min-count 10 sed "s/[^[:alnum:] _-]/ /g" | sed "s/ +/ /g" ---> remove all non-alphanumeric characters dimension: 100/300 window size: 10 min count: 10 iterations: 20
embedding_v3 data: FedGaz + Europarl preprocessing: --lowercase --normalize-digits --min-count 5 sed "s/ [^[:alnum:] _-]+/ /g" | sed "s/ +/ /g" ---> remove all non-alphanumeric tokens (instead of characters) dimension: 100/300 window size: 10 min count: 10 iterations: 20
standard parameters for all of the models alpha: 0.05 iterations: 10 threads: 10 subsampling: 0.001 skip-gram: true HS: false negative: 5
Model Performance Number of correct answered analogies/questions for various models in German.
Model | Syntactic | opposite | bestmatch | doesn't fit |
---|---|---|---|---|
embedding_v1_100 | 24.7% (1797/7270) | 15.3% (46/300) | 14.3% (66/462) | 80.7% (71/88) |
embedding_v1_300 | 29.6% (2149/7270) | 15.7% (47/300) | 24.0% (111/462) | 81.8% (72/88) |
embedding_v2_100 | 26.1% (2058/7897) | 16.3% (49/300) | 22.9% (113/494) | 83.3% (75/90) |
embedding_v3_100 | 22.6% (1810/8022) | 14.0% (42/300) | 17.7% (95/538) | 78.9% (71/90) |
embedding_v3_300 | 31.4% (2518/8022) | 19.0% (57/300) | 32.5% (175/538) | 83.3% (75/90) |
Evaluation examples: opposite: Frage Antwort stark schwach best match: China Yuan Deutschland Euro doesn't fit: August April September Jahr
Incorporated in README
@simon-clematide
Further steps parallel corpus
Further steps multilingual embeddings
<UNK>
symbolFurther steps phrase extraction