Convert numberbatch text file to magnitude format

plasticityai / magnitude

A fast, efficient universal vector embedding utility package.

MIT License

1.63k stars 120 forks source link

Convert numberbatch text file to magnitude format #12

Closed Shubham067 closed 6 years ago

Shubham067 commented 6 years ago

I am trying to convert a "numberbatch-en.txt" pre-trained word vector file to magnitude format using the converter but it's throwing ValueError. Is it only useful for converting glove & word2vec pre-trained files? https://github.com/commonsense/conceptnet-numberbatch screenshot 29 @AjayP13 @acsands13 Please look into this issue.Thanks!

Shubham067 commented 6 years ago

Thanks @plasticity-admin. The issue got resolved now!

AjayP13 commented 6 years ago

Your welcome! I also performed the conversion on Numberbatch and hosted the files here:

Light, Medium, Heavy

in case you run into any other problems, but it should be resolved now.

The issue was we unit-length normalize vectors and Numberbatch has one zero vector in it which is a mathematically degenerate case that cannot be unit-length normalized as it involves a division by zero: