search

avidale / compress-fasttext

Tools for shrinking fastText models (in gensim format)
MIT License
165 stars 13 forks source link

Add description how to load Facebook implementation fasttext model as FastTextKeyedVectors #3

Closed futorio closed 4 years ago

futorio commented 4 years ago

I was trying to load Facebook implementation of ft model from deeppavlov http://docs.deeppavlov.ai/en/master/features/pretrained_vectors.html#fasttext as it written in README.md. But module gives error. _pickle.UnpicklingError: invalid load key, '\xba'.

I solved this problem by loading ft model with function gensim.models.fasttext.load_facebook_model and get FastTextKeyedVectors object:

from gensim.models.fasttext import load_facebook_model
import compress_fasttext
big_model = load_facebook_model('path-to-original-model').wv
small_model = compress_fasttext.prune_ft_freq(big_model, pq=True)
small_model.save('path-to-new-model')

Maybe add this information in README.md?

avidale commented 4 years ago

Done! @futorio Thank you for your feedback. Next time, you can make a pull request of your own ;-)

futorio commented 4 years ago

@avidale Thanks for package