search

ninpnin / probabilistic-word-embeddings

Train and evaluate probabilistic word embeddings with Python.
https://ninpnin.github.io/probabilistic-word-embeddings/
4 stars 2 forks source link

saving large outputs #6

Closed mhbodell closed 1 year ago

mhbodell commented 1 year ago

Trying to save output (e) from "largeish" dynamic model with: e.save(file_name) generate error code:

OverflowError: cannot serialize a bytes object larger than 4 GiB

Suggested solution: change line 108 in probabilistic-word-embeddings/probabilistic_word_embeddings/embeddings.py to "pickle.dump(d, f, protocol = 4)"

Python 3.7.6 Ubuntu 20.04.5 LTS

ninpnin commented 1 year ago

Fixed in v0.14.1 !