Collects all tweets from the sample Public stream using Twitter's streaming API, and writes them to stdout, which you can redirect to a file for later use as a corpus.
The sample Public stream "Returns a small random sample of all public statuses. The Tweets returned by the default access level are the same, so if two different clients connect to this endpoint, they will see the same Tweets."
This module consumes tweets from the sample Public stream and putes them on a queue. The tweets are then consumed from the queue by writing them to a file in JSON format as sent by twitter, with one tweet per line. This file can then be processed and filtered as necessary to create a corpus of tweets for use with Machine Learning, Natural Language Processing, and other Human-Centered Computing applications.
First, you will need to configure the script by supplying tokens that are generated by Twitter for your application.
config-sample.yaml
file to config.yaml
.main()
function.Run the script, redirecting stdout as needed e.g.:
python twitter_corpus.py > tweet-stream.json
Status and error information will be written to stderr.
By default, the script is configured to connect to the sample stream, which "returns a small random sample of all public statuses".
stream.sample()
If you would like to filter tweets by location boxes then be sure to read the location parameter information from the Twitter API. Below is an example to filter tweets for the continental United States.
LOCATIONS = [-124.85, 24.39, -66.88, 49.38,]
stream.filter(locations=LOCATIONS)
If you would like to filter by keywords instead, use the track
parameter. Below is an example to filter for some example emoticons.
EMOTICONS = ">:] :-) :) :o) :] :3 :c) :> =] 8) =) :} :^) "
EMOTICONS = EMOTICONS.strip().split(' ')
stream.filter(track=EMOTICONS)
Please refer to the streaming.py
module from the Tweepy library.
If you would like to modify the application to process tweets as they
are received instead of writing them to stdout, edit the worker()
function as needed.