sloria / TextBlob

Simple, Pythonic, text processing--Sentiment analysis, part-of-speech tagging, noun phrase extraction, translation, and more.
https://textblob.readthedocs.io/
MIT License
9.11k stars 1.13k forks source link

Naive Bayes classifier is slow in classifying for the first time. #362

Open Atharv-Attri opened 3 years ago

Atharv-Attri commented 3 years ago

I am creating a text classification model but for some reason it is taking a very long time for my code to run. I have looked at it in detail and found that the model loads relatively quickly, but the first classification itself takes much longer. The model loads in only 3 seconds, and when I classify for the first time, it takes 13 seconds to classify the text, but then it takes only 0.01. I was wondering if anyone know a way to reduce the time it takes to classify. My code is listed below.

iimport pickle
import time
from textblob import TextBlob
t1 = time.time()
cl = pickle.load( open( "classifier.pickle", "rb" ) )
print("Loading took: ",time.time()-t1)
t1 = time.time()
blob = TextBlob("while x is 1:", classifier=cl)
print(blob.classify())
print("Classifying took: ",time.time()-t1)
t1 = time.time()
blob = TextBlob("x=4", classifier=cl)
print(blob.classify())
print("Classifying took: ",time.time()-t1)
t1 = time.time()
blob = TextBlob("name = 'hello'", classifier=cl)
print(blob.classify())
print("Classifying took: ",time.time()-t1)

This outputs: image

my model code:

with open('model.json', 'r') as fp:
        cl = NaiveBayesClassifier(fp, format="json")
    task = tasks.pop(0)
    console.log(f"{task} complete")
    object = cl
    file = open('classifier.pickle','wb') 
    pickle.dump(object,file)

As you can see, the first classifying attempt takes too long, so is there a way to not have it take that long?

Thanks!