Web-scraping component (first pass)

[wschroder]

First pass of the web-scraping component should do the following:

• Callable from external scheduler component (ref Issue #1 - Create scheduler component)
• Starts up, does it's job, then shuts down when done.
• Keep configuration separate (e.g. don't hard-code URL's, connection strings, etc.)
• Load, crawl, scrape Georgia-state ethics web pages to obtain primary source data (Offices, Candidates, Contributions, Donors)
• Storage options:
  • Dump scraped data immediately as raw, unprocessed data to a "RawData" table for later processing
  • Process & clean up data, loading it directly into the final tables

I'm leaning toward the former (just saving it as raw, unprocessed data to a "RawData" table), so that we can have a separate module which handles the complications that may arise, such as data-cleanup, identifying & filtering duplicate data, etc. (Separation of Concerns)

[bbrewington]

By "raw, unprocessed data" do you mean the raw html?

[wschroder]

By "raw, unprocessed data", I mean the text that is is obtained as output from the web-scraping, so it would not contain any HTML. Example: "Bob Jones, 123 Elm Avenue, $100.00, ..." From a "web-scraping" perspective, it might look processed, but there's more work to be done, e.g. match up the foreign key references, e.g. donor, candidate, etc., and recognize & filter out duplicate data.