WIP: Refactor sense2vec from scratch - Githubissues

explosion / sense2vec

🦆 Contextually-keyed word vectors

https://explosion.ai/blog/sense2vec-reloaded

MIT License

1.62k stars 238 forks source link

WIP: Refactor sense2vec from scratch #77

Closed ines closed 4 years ago

ines commented 4 years ago

Documentation: https://github.com/explosion/sense2vec/blob/refactor/README.md

Use spaCy's Vectors class to store vectors
Make component fully serializable (e.g. with a spaCy model)
Make methods more consistent and add more utilities (to get best-matching sense, all senses etc.)
Improve how most_similar works (average if used with more than one term)
Make package a pure Python package
Add recipes for Prodigy (thanks to @kabirkhan for the initial work on this!)

TODO

[ ] update spaCy version (requires v2.2.2, update to stable once it's released)
[ ] retrain and package vectors
[ ] update training scripts and process

Vectors for testing

I did a quick conversion of the existing Reddit vectors (attached to this release).

Download file from here: https://drive.google.com/file/d/19K2GhJqH40Wz4PtjlnjIUpCBFZhURe15/view?usp=sharing
Unzip and extract directory.
s2v = Sense2Vec().from_disk("/path/to/directory")