Tweet Processing: Cassandra, Spark, Shark - Storage issues

[mstefanro]

NLP, Sentiment/emotion analysis, Clustering etc.

[f4bD3v]

Here is the methodological white paper of the UN global pulse research: http://www.slideshare.net/unglobalpulse/globalpulsecrimsonhexmethodspaper2011

A master thesis on a topic model approach to clustering tweets: "Clustering short status messages: A topic model based approach" http://ebiquity.umbc.edu/_file_directory_/papers/518.pdf

[f4bD3v]

Storing tweets with Cassandra:

Cassandra Query Language (CQL)

Cassandra Data Modelling Best Practices

https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/CassandraCQLTest.scala

https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/CassandraTest.scala

• Counting
  • Count by region
  • Count by city (Delhi, Mumbai, Bangalore, Chennai, Kolkatta)
  • Total count of tweets in India

[f4bD3v]

Issue with Cassandra: too many tweets going through filter