MemoryError when training NaiveBayesClassifier

[cmazzoni87]

I am encountering a memory issue when trying to train my model with large data-sets (140k samples of 200 words each). Is there a way to train the model incrementally?

[Akshay2350]

First do some per-processing and remove unnecessary words from your dataset. In per-processing i would say remove stopwords, punctuation etc. Make it look more similar to each other. Then if possible use online platforms like Kaggle, Google colabs etc. They are cloud platforms with capablity of processing higher system requirement ML models. If still have memory error then go for below solutions.

• Don't load data into model at once, Divide it into batches and then process it.
• Another way is to to use Hadoop. It will allow you to do parallel processing and serialize the data according to your system requirements.

I know its not enough but there is no other way i know to work with big data.