Number of tweets analyzed

[gyachdav]

Fellas, as part of the media blitz we're planning there will be a press release that will throw some big numbers at the readers. Can you provide some impressive statistics about the data your tools processed e.g. our crawler fetched 2M tweets and 10M sentiments keyword processed. Any thing that you think might be interesting IS interesting.

[julienschmidt]

I believe this isn't urgent?

Blocking issues: #42 #47

[julienschmidt]

We'll add an JS API function for that. We just need to add up the values from all characters.

[julienschmidt]

gotsentimental.stats() ⇒ Promise.<Object>

[julienschmidt]

Currently we have about 1.8 million tweets in our database. For about 44% we were able to determine a sentiment (the sentiment detection still needs a lot of work).

[sacdallago]

If it's computing power you are after, let me know

[sacdallago]

Just saying:

[marcusnovotny]

Filthy casual

[julienschmidt]

[sacdallago]

:D That's cute guys... but that was just one of three machines :laughing:

[marcusnovotny]

@julienschmidt We should change the names of our Stats function. Tywin Lannister doesn't have a lot of friends in the fanbase yet ends up on Spot 2 on Most Popular. Grey Worm is universally liked yet number 1 Hated. The scores come from people enthusiastically tweeting about Tywin's death and episodes when Grey Worm was thought to be in great danger.

We should just call it "Highest Positive Scores" & "Highest Negative Scores" and then provide a small explanation on the About Page, what we're meaning with Sentiment Scores. That it's hard to tell wether someone likes a character just from a tweet for us and we're actually measuring how people express their feelings on the events.

Example: "I fucking hate Tywin Lannister, his death serves him right": We measure angriness "I'm extremely glad Tywin Lannister is gone now, he only held his family back": We measure relief

Yay / Nay?

[julienschmidt]

aggregator/aggregator.js#L208 Fix it

[marcusnovotny]

That's why I proposed the new names, I don't know what else to put?

[julienschmidt]

Both popularity and heat should e.g. value recent tweets more than the total

[marcusnovotny]

That still doesn't fix my problem that a positive sentiment score on a tweet doesn't equal that character being popular and vice versa

[julienschmidt]

Depends on your definition of "popular". I think a lot of positive mentions is a valid definition.

We could also include how positive / negative sentiment is.

[julienschmidt]

After all our goal is to show such surprises. If they don't like a character, maybe they should tweet about it...

[marcusnovotny]

Well I guess you can see it that way too. When you check why Tywin Lannister is popular and see that the positive spike is from the episode he dies in, it's actually quite funny :smile:

[gyachdav]

Now that all is said and done. can you give us some numbers? examples:

• [ ] number of tweets
• [ ] languages
• [ ] peak season/epsiode
• [ ] low season/episode
• [ ] average per episode
• [ ] in total are we seeing more negative or positive tweets?

[gyachdav]

Status?

[marcusnovotny]

Hey Guy!

Our db wasn't completely exhaustive. I'm currently crawling missing characters and am at around 2.75 million tweets with 1 million analyzed, numbers still rising. Of those 2.75 million, we were able to analyze 1 million: 600k are postivie, 420k negative. I'll provide a dump for Project F as soon as I'm done.

Peak season / episode varies for each character. If a character is only active in season 2-3 for example, he gets mostly mentioned then. The most extreme peak is for Jon Snow at the season 5 finale though. Apart from spikes like that, normally the number of tweets rises with each season, since more and more people join Twitter I suppose

[sacdallago]

Hey guys! Wasn't there a megadatabase going around somewhere? Something I could use on the www.got.show page?

[marcusnovotny]

Currently crawling the very last characters. Hope to have it up by tonight, tomorrow the latest

[sacdallago]

Nice!!

[marcusnovotny]

DB dump coming tomorrow afternoon. Pure excitement :grin:

[marcusnovotny]

Just spotted Lord Varys and Khal Drogo on the blacklist. Currently crawling those, afterwards I'll upload the dump. 2 hours max I guess

[marcusnovotny]

Still crawling Drogo, I fear there's a load of Spanish tweets in there. Let's see how it turns out. Update definitely coming today.

[sacdallago]

:zzz:

[marcusnovotny]

Still on it. Daenerys had a hole in the data. Think I'll drop Drogo, there seem to be a lot of unrelated tweets about him. Will upload afterwards.

[sacdallago]

K :) Just remember to give me the db access later :) And tomorrow we can try it out on www.got.show

[sacdallago]

P.S.: I'll go to bed now coz I'm dead, so don't expect me to answer before tomorrow at some stage :P noonish

[marcusnovotny]

If you don't need it tonight I might crawl Drogo too

[sacdallago]

Nah, take your time! I mean, I'm still waiting for so many other things that I'm an induced procrastinator now.

[marcusnovotny]

That's the spirit.

[sacdallago]

Yeah, and to think that I used to have a life...

[marcusnovotny]

Conceal, don't feel, don't let them know :broken_heart:

[julienschmidt]

Included in the Report