search

plasticityai / magnitude

A fast, efficient universal vector embedding utility package.
MIT License
1.63k stars 120 forks source link

Normalization #22

Closed alexpnt closed 6 years ago

alexpnt commented 6 years ago

First of all: Nice work in implementing useful tasks such as lazy loading and memory mapped files .

As a suggestion, I think it would be nice to have an option regarding the normalization. I might need the vectors as they are and not normalized.

AjayP13 commented 6 years ago

Thanks!

This is coming! Unfortunately, it will take some time as I will have to re-convert all the pre-converted models before releasing this release. I will ping you on this issue when it comes out.

AjayP13 commented 6 years ago

Sorry, this took longer than normal. It required a re-work of the library and file format since when I created this I didn't actually encode the non-normalized vector information in the files.

The constructor now takes a normalized parameter that you can use to turn normalization on and off. It is documented here. Note: You will have to re-download the .magnitude file for the model you are using or re-convert it if it is your own for this to work. You can download them again from here.