Export of stream tweets to csv.

[justinlittman]

Stream tweets are currently stored in time segmented line-oriented JSON files. Support for exporting these tweets to CSV is required. This support should include:

• Optionally retaining the existing time segmented files or merging into fewer files. If merging, it should be possible to specify the maximum number of tweets per file.
• Optionally sampling the tweets at a specified rate.
• Counting the tweets (necessary to determine sampling rate).
• Limiting by time (based on time segmented files).

[justinlittman]

Here's info from @kerchner on how he has done this manually in the past:

@Justin, on second thought it looks like I did write a script to do just that.  Step 1 was "gunzip -r" the tree of twitterfilter output files.  Step 2 was "grep -h <twitterfilterdirectory> -r > singleoutputfile.json".  Then the python script, which apparently I didn't place in a good location, to read each line's json and extract the relevant columns and write a line of csv.  I'll see if I can find it but it wasn't too complicated.
Ah, here it is:  https://gist.github.com/kerchner/e471a5f010b6bcafb3f0