Memory leak on corrector.LoadLangModel()

bakwc / JamSpell

Modern spell checking library - accurate, fast, multi-language

https://jamspell.com/

MIT License

612 stars 103 forks source link

Memory leak on corrector.LoadLangModel() #44

Open rominf opened 6 years ago

rominf commented 6 years ago

I've trained my model (I've tried versions from master and 0.0.11 branches) on 10 MiB plain text part of English Wikipedia (enwiki-latest-pages-articles_10MiB.txt) and got 41 MiB bin file (enwiki.bin.zip).

I'm loading it in Python, but it takes 12 GiB of memory to load it and still it doesn't load in foreseeable time.

xdingo93x commented 6 years ago

I'm facing the same issue. I've trained a (german) model using around 560MiB plain text from the Leipzig Corpora Collection. The model itself is 488MiB. Having a 16GB RAM 4CPU Linux Cloud, it takes 5-10 min to load the model. Is it possible to speed this up?

mshakirDr commented 2 years ago

March 16, 2022 I tried to use this on Windows 10, Python 3.9.7 but it keeps eating memory and nothing happens. Unable to load the 34 MB English model.

bakwc commented 2 years ago

Community version doesn't support loading linux models on windows. You should either use a linux or train your own model on windows.

Or you can buy a PRO version, it supports all models on all operation systems. Also a PRO version reduced memory usage while training.