Open aronwc opened 8 years ago
We definitely need to reduce vocabulary size using min_df (currently 1,509,078 terms)
Depending on how this goes, we may also consider a slightly different prediction problem. E.g., a regression where the input is a tweet and the output is how many days prior to first protest tweet. While this is insanely hard to predict, it may capture the idea that the intensity of certain terms should increase over time.
Even though the window=50 has the biggest testing accuracy, the best model in my view, is the one with a window=10 because the positive coefficients are the most correlated to the protests, for example:
negative_window | ngram | min_df | max_df | train acc | test acc | n users |
---|---|---|---|---|---|---|
10 | (1,2) | 2 | 1.0 | .591 | .507 | 285 |
20 | (1,2) | 4 | 1.0 | .611 | .520 | 274 |
30 | (1,2) | 4 | 1.0 | .617 | .513 | 264 |
40 | (1,1) | 1 | 0.1 | .837 | .537 | 259 |
50 | (2,2) | 2 | 0.1 | .751 | .559 | 255 |
I played around with a slightly different classification setup. (See Classification-Optimize-Window. )
I introduced two parameters:
window_sz
: how many tweets should be used to construct each instance.gap_sz
: how many tweets should there be between the final tweet of the negative instance and the first tweet of the positive instance.For example, if the first protest tweet occurs at tweet number 100, window_sz=5
, and gap_sz=10
, then we'll get:
tweets[95:100]
tweets[80:85]
The main difference is that previously the positive and negative instances contained many of the same tweets, whereas here they have no tweets in common.
I then iterated over many, many combinations of tokenizations, window_sz, and gap_sz. The results are here. The top 20 results are:
0.595926 collapse_mentions=True collapse_digits=True binary=False ngram_range=(1, 2) min_df=2 use_idf=False norm=None window_sz=1 gap_sz=20
0.593526 collapse_mentions=True collapse_digits=True binary=True ngram_range=(1, 1) min_df=4 use_idf=False norm=None window_sz=20 gap_sz=100
0.590641 collapse_mentions=True collapse_digits=False binary=False ngram_range=(1, 2) min_df=2 use_idf=True norm=None window_sz=20 gap_sz=100
0.590471 collapse_mentions=True collapse_digits=True binary=True ngram_range=(1, 1) min_df=2 use_idf=False norm=None window_sz=1 gap_sz=20
0.590404 collapse_mentions=True collapse_digits=False binary=True ngram_range=(1, 2) min_df=4 use_idf=False norm=None window_sz=1 gap_sz=20
0.588552 collapse_mentions=True collapse_digits=False binary=True ngram_range=(1, 1) min_df=2 use_idf=False norm=None window_sz=1 gap_sz=20
0.588519 collapse_mentions=True collapse_digits=False binary=True ngram_range=(1, 1) min_df=4 use_idf=False norm=None window_sz=1 gap_sz=20
0.588462 collapse_mentions=True collapse_digits=False binary=True ngram_range=(1, 1) min_df=2 use_idf=False norm=None window_sz=20 gap_sz=100
0.588397 collapse_mentions=True collapse_digits=True binary=False ngram_range=(1, 2) min_df=2 use_idf=True norm=l2 window_sz=20 gap_sz=100
0.588095 collapse_mentions=True collapse_digits=True binary=True ngram_range=(1, 1) min_df=4 use_idf=True norm=None window_sz=5 gap_sz=100
0.588095 collapse_mentions=True collapse_digits=False binary=False ngram_range=(1, 1) min_df=4 use_idf=False norm=None window_sz=5 gap_sz=100
0.586835 collapse_mentions=True collapse_digits=True binary=True ngram_range=(1, 2) min_df=4 use_idf=True norm=l2 window_sz=1 gap_sz=20
0.586768 collapse_mentions=True collapse_digits=True binary=True ngram_range=(1, 2) min_df=2 use_idf=False norm=None window_sz=1 gap_sz=20
0.586667 collapse_mentions=True collapse_digits=True binary=False ngram_range=(1, 1) min_df=4 use_idf=False norm=None window_sz=1 gap_sz=20
0.585123 collapse_mentions=True collapse_digits=False binary=True ngram_range=(1, 2) min_df=4 use_idf=False norm=None window_sz=5 gap_sz=20
0.584983 collapse_mentions=True collapse_digits=False binary=True ngram_range=(1, 2) min_df=2 use_idf=False norm=None window_sz=1 gap_sz=20
0.584949 collapse_mentions=True collapse_digits=True binary=False ngram_range=(1, 1) min_df=2 use_idf=False norm=l2 window_sz=1 gap_sz=20
0.584038 collapse_mentions=True collapse_digits=False binary=False ngram_range=(1, 1) min_df=4 use_idf=True norm=None window_sz=50 gap_sz=50
0.583462 collapse_mentions=True collapse_digits=False binary=True ngram_range=(1, 1) min_df=4 use_idf=True norm=l2 window_sz=20 gap_sz=100
0.583397 collapse_mentions=True collapse_digits=True binary=True ngram_range=(1, 1) min_df=2 use_idf=False norm=None window_sz=20 gap_sz=100
I recommend we stick with the third place one:
0.590641 collapse_mentions=True collapse_digits=False binary=False ngram_range=(1, 2) min_df=2 use_idf=True norm=None window_sz=20 gap_sz=100
This has high accuracy, uses a largish window size (2), and also includes bigrams.
Here are the top terms. You'll have to tell me if they make any sense ;)
X.shape=(398, 12198)
HASHTAG_sabadodetremuranosdv
HASHTAG_globonews
ele
bem
volta
ontem
seu
THIS_IS_A_URL rt
cruzeiro
falando
photo
sim
dilma
que eu
joga
dia do
esse
chegando
eh
p
tudo
calma
o o
amiga
campos
parabéns
ahahahahaha
MENTION ahahahahaha
presidente
you
amor
alguns
perfil
MENTION te
MENTION HASHTAG_globonews
casa
cabeça
tem
pastor
HASHTAG_verdadessecretas
If I use that setting and vary the gap size, I get the following graph:
While there is some variance, accuracy generally increases with gap size, as expected.
Based on this, I recommend the following:
So, for the testing set is going to be all tweets a user have posted in jumping windows of size 20, it does not go just up to time T correct?
Up to the first protest tweet.
On Fri, Oct 16, 2015 at 8:55 PM, ElaineResende notifications@github.com wrote:
So, for the testing set is going to be all tweets a user have posted in jumping windows of size 20, it does not go just up to time T correct?
— Reply to this email directly or view it on GitHub https://github.com/tapilab/protest/issues/17#issuecomment-148876386.
I have changed Classification-testing nb and created a list of stopwords for Portuguese language and added it to default English list of stopwords. After this, we have a better list of coefficients. Such as:
HASHTAG_sabadodetremuranosdv
THIS_IS_A_URL rt
HASHTAG_sextadosdvcomsrtwitteiro
perfil
###### dilma
photo
volta
ontem
###### pt ---> dilma's party
sábado
chegando
cruzeiro
globo
brasil
sim
THIS_IS_A_URL acabou
THIS_IS_A_URL
falando
15
p
###### HASHTAG_debatenaglobo ---> debate on globo channel
tipo
###### corrupção ---> corruption
amiga
passa
bem
mudar
joga
preguiça
gol
parabéns
vamos
sábado 15
###### fora ---> out
###### presidente ---> president
HASHTAG_brasil
campos
gosta
HASHTAG_verdadessecretas
t
gols
photo THIS_IS_A_URL
status
metade
fred
###### deputado ---> deputy
vocês
casa
amor
###### panela ---> pot
HASHTAG_luciananaglobo
HASHTAG_mudabrasil
MENTION te
cortina
posted photo
uso
THIS_IS_A_URL THIS_IS_A_URL
###### vergonha ---> shame
HASHTAG_aécio45
publicar
de…
vez
te
marina
MENTION sim
calma
dom
###### manifestação ---> manifestation
amanhã
propaganda
fifa
cabeça
nós
###### band ---> news channel
hoje
###### petrobras ---> the biggest company which sells gas, etc http://www.petrobras.com/en/home.htm
###### HASHTAG_panelacohoje ---> hit on the pots today
vontade
pessoas
hahahaha
###### petistas ---> people from dilma's party
pe
feliz
lua
pão
###### sbt ---> news channel
contigo
HASHTAG_quintacomvalentinonosdv
desistir
##### protesto ----> protest
MENTION amiga
bater
##### partido --> party
whats
brother
MENTION MENTION
aula
THIS_IS_A_URL photo
rede
ato
I tested several configurations checking their vocabulary but at the end I used the same configuration you advised. I could get accuracy of 60% with gap_sz=150, but I prefered the vocabulary using 100.
For testing set: I changed it to read a list of tweets up to first to use some keyword and then I read the list starting at the end of it, joining each 20, making groups and jumps like we previously discussed.
Here are the plots for testing instances: see Plots/
PS.: I am getting just 275 users in total at testing time, I am checking why.
Our final model at this part is:
Model characteristics: collapse_mentions=True, collapse_digits=False, binary=False, ngram_range=(1,2), min_df=2, use_idf=True, norm='l2', window_sz=20, gap_sz=100
In the initial tests this model accuracy was 0.575769, after adding stopwords we get 0.588205.
Some users' first post comes with one of the keywords, so they are not considered, then we have at the end of testing time, 312 users.
Nb Classification-Optimize-Window has precision, recall, f-score and a confusion matrix for the model.
In Classification-testing I added the feature of neighbors hashtag use in the method iterate_instances_changed and I yield the result. How can I proceed in order to add this feature to our model?
How can I proceed in order to add this feature to our model?
It looks like iterate_instances_changed
returns the neighbor feature in the iterator. What you'll need to do is store that result in a list (like is done for users: x[0] for x in iterator if not users.append(x[2] and not neighbors.append(x[3])
)
Then, you'll have to append a new column to the X matrix returned by the vectorizer. Something like this:
>>> from scipy.sparse import hstack
>>> X = hstack((X, np.array([neighbors]).T)).tocsr()
I did it like you advised. But for fit_transform I get one column more, and when I try to use predict_proba for the testing set I get an error because of the difference of dimension for columns. Did I do anything wrong?
ValueError: X has 6850 features per sample; expecting 6851
Make sure that:
On Tue, Oct 20, 2015 at 8:16 PM, ElaineResende notifications@github.com wrote:
I did it like you advised. But for fit_transform I get one column more, and when I try to use predict_proba for the testing set I get an error because of the difference of dimension for columns. Did I do anything wrong?
ValueError: X has 6850 features per sample; expecting 6851
— Reply to this email directly or view it on GitHub https://github.com/tapilab/protest/issues/17#issuecomment-149750839.
When I consider just the neighbors feature for vectorizing I got an error (tokenizer =None, stopwords=None, lowercase=False):
TypeError: expected string or buffer
What I did to get it to work is to change the feature to string, but I believe it is not correct. With this approach the result of vocabulary is:
[('16666666666666666', 0.40124891749570202)]
What should I change to have the features as float?
To use just the neighbor features, you do not need to run the vectorizer.transform method. Instead, just create a 2d array with the neighbor float values. E.g., [[.40], [.12], ...]
On Fri, Oct 23, 2015 at 7:06 AM, ElaineResende notifications@github.com wrote:
When I consider just the neighbors feature for vectorizing I got an error (tokenizer =None, stopwords=None, lowercase=False):
TypeError: expected string or buffer
What I did to get it to work is to change the feature to string, but I believe it is not correct. With this approach the result of vocabulary is:
[('16666666666666666', 0.40124891749570202)]
What should I change to have the features as float?
— Reply to this email directly or view it on GitHub https://github.com/tapilab/protest/issues/17#issuecomment-150555419.
That is true! I am not thinking :( . I am so sorry for the dumb question.
No worries! If this stuff were easy, I would be out of a job.
I checked the model coefficient for different windows' sizes because the smaller the window is, less neighbors we have (different of 0) we have. What I understand from these coefficients is that the neighbors feature is important for our model and we can use it .
For clustering I am analysing different numbers and the centroids. I found interesting that the user Ary_AntiPT is in all different clusterings, this user's name means Anti PT (dilma's party) and he has 163 neighbors.
UPDATE:
Plots for each cluster are in Clustering Plots
File name example: clf3_0.png, where 3 is the quantity of clusters and 0 is the number of the cluster (0,1 or 2 in this case)
Great! I think the Clustering Plots are numbered in reverse order? E.g., clf8_7 seems to have 9 users, which would be the first row in the 8 clusters table.
Could you please also create three additional plots showing just the centroids of each cluster? (e.g., for 3 clusters, you would have one plot just showing cassiosalva, Moreira and Ary).
Sorry, the table was not in order. I updated it now. New plots you have asked are in Centroids. plots: 3 clusters, 6 clusters and 8 clusters.
And I added plots for each user(centroid) alone.
For each model of cross-validation I used the metrics of precision, recall and f-score and I calculated the mean and standard deviation for all 10 models:
I am analysing the dendrogram below created with our data. Based on it we can create 4 or 5 clusters. I am still working on it and I hope find some interesting info from these groups.
Precision, recall and f-score correction: