LDA with mallet: inspect memory/time on larger corpora

GateNLP / gateplugin-LearningFramework

A plugin for the GATE language technology framework for training and using machine learning models. Currently supports Mallet (MaxEnt, NaiveBayes, CRF and others), LibSVM, Scikit-Learn, Weka, and DNNs through Pytorch and Keras.

https://gatenlp.github.io/gateplugin-LearningFramework/

GNU Lesser General Public License v2.1

26 stars 6 forks source link

LDA with mallet: inspect memory/time on larger corpora #78

Open johann-petrak opened 6 years ago

johann-petrak commented 6 years ago

The LF spends a lot of time and uses a lot of memory after the topic words have been logged to the message pane.

Figure out which steps are slow and maybe make them optional (could be the coherence calculation).

Also, there may be a memory leak somewhere: re-running caused an out of memory error inside the gui