Facebook Scraping

[shiffman]

I created a quick "scraping facebook" example to grab all the comments by date from https://www.facebook.com/barackobama, issues are as follows:

• [ ] I quickly exceeded my rate limit:

  { error: 
  { message: '(#17) User request limit reached',
   type: 'OAuthException',
   is_transient: true,
   code: 17,
   fbtrace_id: 'FvvmJfY5prt' } }

• [x] It would probably make more sense to start by scraping the actual page content.
• [ ] I'm not sure how to best detect when everything has completed given the recursive nature of the pagination.

[shiffman]

Here is a list of "scrapers" that would be nice.

• [x] scrape all posts for a page, along with all metadata
• [ ] download all images / video associated with any given post?
• [ ] scrape all comments to each post.

Here are the pages that would make sense to scrape:

• [x] https://www.facebook.com/barackobama/
• [ ] https://www.facebook.com/WhiteHouse
• [ ] https://www.facebook.com/michelleobama/
• [ ] https://www.facebook.com/VicePresidentBiden/

Also. . .?

• [ ] https://www.facebook.com/DonaldTrump/
• [ ] https://www.facebook.com/theapprentice/

[shiffman]

With 2b888c9e3573cb0afc690c796c9a2398397a040d and 74272b37299eb30847e76eb427fcd8153499d348 I now have a working facebook scraper. I believe I've scraped everything from BarackObama and WhiteHouse (see /data for resulting files), but would be good to doublecheck. Likes, comments, etc. would need further scraping as the code I wrote only takes the first "page" of each of these with each post.