search

TutteInstitute / vectorizers

Vectorizers for a range of different data types
BSD 3-Clause "New" or "Revised" License
97 stars 23 forks source link

Serialization #62

Open stevemarin opened 3 years ago

stevemarin commented 3 years ago

Have you thought about adding serialization/saving to disk? At the very least, would you point out what to store for, say, the ngram vectorizer?

Thank you very much for the examples on training word/document vectors using this and comparing to USE! Giving it a try now on my data and it looks pretty good! Thank you!

stevemarin commented 3 years ago

Does this portion capture what's needed for the ngram vectorizer?

https://github.com/TutteInstitute/vectorizers/blob/master/vectorizers/ngram_vectorizer.py#L269-L276

stevemarin commented 3 years ago

Sorry for all the comments! Pickling the ngram vectorizer seems to work as long as all the numba attributes in the class are converted back to regular Python objects. Does this agree with your understanding?

lmcinnes commented 3 years ago

Yes -- to properly handle things we really need to set up the __getstate__ and __setstate__ methods to properly convert the numba based attributes back and forth. Usually this isn't too hard, but it is something that we haven't gotten to yet. With that done pickling should just work. I'll add an issue for that so that we hopefully don't forget it.