Testar modelos

[tatchiwiggers]

Professors suggestion: use CountVectorizer and TF-IDF;

For CountVectorizer (Bag of words):

• take text and break into individual tokens;
• encode tokens;
• count how many times each word appeared in the text;
• call transform on CountVectorizer instance;
• check that the text has been encoded into sparse matrix;
• convert this sparse matrix into numpy array;
• convert it into pandas dataframe;
• run _ngramrange to indicate how many words will be considered together for tokenization;
• countVectorizer considers single-word to four-word combination as separate token. Add vocabulary option to it to meet the requirement;

For a TF-IDF Model:

• Tokenize;
• Find TF-IDF Values value for each word in the sentence;
• calculate TF-IDF values by multiplying TF values with their corresponding IDF values;
• sort the dictionary in the descending order of the frequency;
• filter the most frequently occurring words;
• find the TF-IDF values for all the words in each sentence;
• consider using max_df, min_df, max_features and ngram_range;

For a Random Forest Model

For a Naive Bayes Model:

The Naive Bayes algorithm is a supervised machine learning algorithm based on the Bayes' theorem. It is a probabilistic classifier that is often used in NLP tasks like sentiment analysis (identifying a text corpus' emotional or sentimental tone or opinion. HOWEVER, Naive Bayes Model works particularly well with text classification and spam filtering.

[tatchiwiggers]

Modelo NÃO estpa overfitting, porém foi testado com os labels incorretos gerando resultados imprecisos. CONSERTAR.

[tatchiwiggers]

Modelo RNN Testando em média 85%. 👯‍♂️ Preciso fazer ele rodar corretamente com nosso dataset que depois de processado agora está testando 100%.... 😥

[tatchiwiggers]

Modelo de Regressão Linear Após o "re-preprocessamento" do dataset está testando em média 88% nos três sentimentos: NEUTRAL, POSITIVE and NEGATIVE.

[tatchiwiggers]

Modelo RandomForestClassifier testando somente 39% 😥

[tatchiwiggers]

Decido pelo modelo pré-treinado VADER. Vamos adicionar alguns features para dar mais precisão à análise, como tradução de emojis e algumas gírias. 🤖