Is it possible to host model file in some database and query it's results?

[vondiplo]

Do you know of any project aiming to support saving and loading wiki2vec or word2vec model files into a database (like mysql) and provide some adapter to query the vectors ?

[dav009]

not sure if it is exactly what you are looking for, but I've seen lots of people using: https://github.com/3Top/word2vec-api probably they still load it on memory if that's what you are trying to avoid

[vondiplo]

Thanks for this awesome response, though it's not exactly what I meant. Both solutions require loading the entire model file into the computers internal memory. This is a bit overwhelming. Do you know if it's possible to save the model file into a database somehow (that is persisted on disk) and query the database for the words and vectors?

[dav009]

• are you currently able to load them into memory? if so you can dump them into a flat format (tsv) I got a gist for that somewhere.
• if you are using a language like java you could use a memory-mapped datastore (check ChronicalMap) this means your hash from words to vectors would live in disk and only certain part of it gets loaded into memory as needed. I've worked on projects using this approach and it is quite fast, while allowing you to have a hash with very long vectors.
• if you are using python, you could use a sqllite db and numpy arrays. Maybe something similar as suggested in this question, I'm not sure how efficient this is.

[dav009]

this is the script for dumping them into a plain file: https://gist.github.com/dav009/11ebe5467419857e054bd9af55090b1b

[vondiplo]

@dav009 - Thanks!!! I've got one last question which I guess is only somewhat related so I'll open it in a new thread.