Scraping Facebook Groups Posts

[dragosrotaru]

Note on FB Scraping, Data Privacy, Future Roadmap

See #5

Pre-requisite: Seed Data

See #6

Requirements

• Don't commit data, private info, credentials, etc.
• write your script in a new folder "scripts/facebook-group-posts-scraper
• use any language you want. Preferably Python or NodeJS/TypeScript. (please)
• use conservative rate-limiting and a dynamic DOM renderer like selenium or Puppeteer.
• the script should get FB_USERNAME, FB_PASSWORD + MongoDB credentials via .env file
• the script should get group ids from #6 (see comments)
• the script saves data to local MongoDB instance (see schema below)

Scraping Facebook Groups Posts

We will use this data to make a news aggregator and to keep an eye out for more data for coalition-building purposes.

I started a script in scripts/facebook-group-posts-scraper using this library: https://github.com/kevinzg/facebook-scraper

It works ok, but it doesn't work with 100% consistency, you will have to troubleshoot and maybe edit the script.

How your script will store and normalize the data

Database will be MongoDB

Schema

type Post = {
  id: UUID,
  createdAt: TimeStamp,
  text: string,
  link: URL,
  likes: number,
  shares: number,
  comments: number,
  groupID: UUID,
  scrapedAt: TimeStamp,
  scrapeID: UUID,
}

Misc

Random Lib I found: https://github.com/ParvJain/Facebook-Group-Scraper (please look through)