Open Ishitori opened 5 years ago
Your guess is correct, the minimum ngram size in the pre trained model has length 3. So for words with less than 3 characters no vectors can be obtained. The fasttext model is not particularly helpful for Chinese in that sense as most words have less characters. For Chinese, character ngrams should rather be computed over radicals (but that is not what fasttext does). Would you be interested in working on some radical based model?
I see. Well, why does it fail then? I would expect to receive an empty vector for that case.
Well, the assumption was that you would probably have been more surprised to find some 0 vectors, not knowing why, than to realize that there is a problem thanks to a KeyError
.
The KeyError
just shows that fastText is not sufficient for the use-case. With a different embedding method, e.g. http://www.cse.ust.hk/~yqsong/papers/2017-EMNLP-ChineseEmbedding.pdf such error would not be raised.
Nevertheless, it would be nice to make the behavior configurable. I propose to generalize the current unknown_lookup
in TokenEmbedding to a general backoff
functionality and make it easily configurable. For your expectation to be met, you would then assign some zero_vector_fn
as the backoff method of fastText. But it is currently not possible.
import mxnet as mx import gluonnlp from gluonnlp import Vocab, data
embedding_model = 'cc.zh.300' # or 'wiki.zh' zh_embedding = gluonnlp.embedding.create('fasttext', source=embedding_model, load_ngrams=True)
def pad_token(token): if len(token) < 3: return token + '@@' else: return token
tokens = ['猴姆'] padded_tokens = [pad_token(token) for token in tokens] cntr = data.count_tokens(padded_tokens) vocab = Vocab(cntr) vocab.set_embedding(zh_embedding)
print(vocab.embedding[padded_tokens[0]])
I get
KeyError
exception if I usecc.zh.300
embedding withload_ngrams=True
and provide a token which has no default embedding.My expectation is that for an unknown token a ngrams strategy would kick in and I would get a combination of vectors. But maybe due to the size of the token (only 2 characters) or maybe because I use specifically
cc.zh.300
embedding I get an error.The minimum reproducing example is below:
Exception is:
If I change token, for example by adding an exclamation mark to the end, to '猴姆!' the error disappear. The problem doesn't appear as well, if I change embedding to 'wiki.zh'.
GluonNLP version is '0.4.1'.
@leezu