search

avidale / compress-fasttext

Tools for shrinking fastText models (in gensim format)
MIT License
165 stars 13 forks source link

load model issues #12

Closed ldb999 closed 2 years ago

ldb999 commented 2 years ago

I trained a Korean language unsuperviesd model and was going to use compress_fasttext load model, but got a error: invalid load key, '\xba'. It seeams ues wrong encoding when load model.

error message: 1460 with open(fname, 'rb') as f: -> 1461 return _pickle.load(f, encoding='latin1') # needed because loading from S3 doesn't support readline() 1462 1463

UnpicklingError: invalid load key, '\xba'.

avidale commented 2 years ago

Hi @ldb999, can you load the original model into pure gensim, without compress-fasttext? If the same error persists with gensim, then please raise an issue in their repo. And if the error does not occur with gensim, but does occur with compress-fastetxt, then it is really on our side. In this case, please give us the code that can reproduce this error, and the link to the model file with which it happens.

ldb999 commented 2 years ago

Thank you for your answer! I got same error in gensim, it seems issue from gensim.