Groups support

[ginatrapani]

Capture all the lists owner’s Twitter users are on, display this somehow (tag cloud?) Sort replies by list membership (show me only replies by people on “my best friends” list)

Here's a rundown of what has to be done:

1. Create 3 new database tables: groups, group_owners, group_members. The groups table will be keyed on group_id and the fields will represent all the metadata for individual groups, plus a network field that defaults to the value 'twitter.' The group_members should just be two fields: group_id and member_user_id. The group_owners table should also be two fields: owner_user_id and group_id. The new table creation SQL should go in both build_database.sql as well as a file in the db_migrations folder.
2. The API call the crawler should use is: http://apiwiki.twitter.com/Twitter-REST-API-Method:-GET-lists Add it to the array of API calls in /common/ class.TwitterAPIAccessorOAuth.php in the prepAPI method. Then, add the list XML parsing case to that same class's parseXML method.
3. Create a Group object and GroupDAO object in a new /common/ class.Group.php file. You can use the class.Post.php file as an example.
4. Add a fetchGroups method to the Crawler class which makes the API call and parses the XML, then uses the GroupDAO to save the data to the right tables.
5. Add $crawler->fetchLists() to the /thinktank/common/plugins/twitter/twitter.php script in the twitter_crawl() method and voila! We'll be crawling lists.
6. Write tests for the Group DAO's and the new TwitterCrawler method.

This sounds like a lot but it's not that bad if you look at the existing code; it's exactly how every other type of data is retrieved (posts, friends, followers, etc).

Something to keep in mind: The growing/shrinking problem applies to groups as well as posts and favorites. I'd want ThinkTank to reflect the current state of a group, so maybe once every X days the crawler will have to wipe group data and recapture it.

[jrunning]

I've started work on this, FYI, over here: jrunning/thinktank@c1b60022ce2d62923f1c0c35e25b08ec4b313034

[jrunning]

P.S. Don't pull that, btw. I haven't even run it through PHP yet to check for syntax errors, much less correct functionality.

[ginatrapani]

Woot! Lookin' good. Thanks Jordan!

[jrunning]

Hah. PHP disallows 'List' as a class name (it's a reserved word) so I'm going with MemberList for now. Making good progress. Also did some refactoring in TwitterAPIAccessorOAuth.

[ginatrapani]

Instead of MemberList, can we name it Group? It doesn't really matter--just semantics, really--but we don't use the word member anywhere else, and Facebook calls lists of users groups, so at least it applies there.

[jrunning]

Yes, that makes sense. I'll switch it in my next push.

[cwarden]

I've done most of the work to implement this feature in 0be9fd8f47f17c6476777555ee88f96ca22df25d. I record the daily group membership count like follower_count since that's the one metric I need right away.

I still need to handle the removal from a user from a group and the deletion of groups, and I need to write unit tests.

[ginatrapani]

Fantastic, thanks Christian! Send on a pull request when you've got tests written and I'll do a full code review.