search

cbaziotis / ekphrasis

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
MIT License
661 stars 90 forks source link

Memory usage #9

Open xro7 opened 5 years ago

xro7 commented 5 years ago

As i noticed there is no trivial way to free memory allocated in RAM after creating a TextPreProcessor object. I tried remove object reference with: preprocessor = None or del preprocessor but it does not seem to work. Is there something i am missing?

cbaziotis commented 5 years ago

The library uses memoization. You can either shrink the maxsize, or call python's garbage collector manually.

import gc

# ekphrasis preproccessing...

gc.collect()