Which fields to include in the summary files

[computermacgyver]

Currently, the summary files have:

• created_at --- UTC timestamp of when the tweet was sent
• id -- a unique number from Twitter identifying this tweet
• tweet_lang -- Twitter detected language of the tweet (not always accurate and not always available)
• user_id -- a unique number from Twitter identifying the author the tweet
• user_created_at --- the UTC time the user created his/her account
• user_lang --- the user interface language the user has set
• location -- a free text field the user can fill out in response to "Where are you located?"
• statuses_count -- number of tweets the user has sent to date
• time_zone --- e.g., US Eastern
• utc_offset -- e.g., UTC-05
• text --- the full text of the tweet (extended_text if the tweet is over 140 characters)

Questions:

• Are there any fields to delete?
• Are there any fields to add (see documentation)
• Do we want to include the full text of tweets? This adds significantly to the file size, but obviously seems useful. Do we want to delete the text from the non-summary files to save space?

Note:

• I will add a separate detected language field that will cover all the data, but have not yet done this (See #1 )

[computermacgyver]

In addition to language detection fields, a "retweet" field is needed. This should be NA if a tweet is an original tweet. If a tweet is a retweet, then this field should contain the numeric id of the original retweet.

We may also want to expand links and have this as an additional field.

[computermacgyver]

Add "geotag" field. This should be coordinates if present, but otherwise place if present. If neither coordinates nor place is present this can be null. (Use "NA" for compatibility with R?)

[ThePrincipleMan]

Hello there Found out about this from red hen labs barnyard and found it interesting. Some points we may think of are

• Secondary geotag of what place(s?) is being mentioned in the tweet.
• A directional graph mapping relations between users with weights(+ve for +ve replies/retweets and likewise for -ve)
• A data structure for an event if it is mentioned more than x number of times in tweets, will be useful for tagging its contents later on and give place the tweet in appropriate baskets
• Uncanny similarities of content (somewhat like plag checker in Competitive Coding sites) will help identify if tweet was copied from other sources giving us a possible way to identify coordinated information dissemination. ... and many more, would really like to hear your thoughts on above.