Refactor for custom scraper settings

[captjt]

Premise of Refactor

Prototype Crawler (ETL) is working but because we want to have an easy way to reproduce these custom crawlers and scrapers ...

... we need to modularize the code to be reusable packages that we can customize to our wishes

[captjt]

Customizable Settings

• [x] Specific elements we want to scrape #580 ~Curate a list of possible elements to scrape~
  • [x] Modularize the scrape function to take multi parameters for nodes
• [x] Environment settings for a specific crawler/scraper/elasticsearch instance
• [x] Check URLs for nested links within themselves (something.com & something.com#top, etc)
• [x] Add tagging from subdomaining for POC filtering in queries #575
• [x] Allow for top level domain checks for urls (.com, .gov, etc.) #584
• [x] Add time stamping based on when pages were indexed #591
• [x] Add logging #600

Scraping custom elements

Creating a modular way to do custom scraping through settings is a bigger lift will have to have specific ways we want to drill into html nodes.

[captjt]

Benefits

Bi-product will be multiple package files and an API to use ... super simple and super powerful

[captjt]

From 01-09

• Normalize and check URLs for nested links within themselves (something.com & something.com#top, etc)
• Add tagging from subdomaining for POC filtering in queries

[captjt]

From 02-09

• Needed to be able to toggle logging on/off on runs so we can audit everything and track where the traffic is going and track everything we are indexing

[captjt]

From 02-15

• Refactor the Runner to be a type that can be concurrent for multiple root sites to be ran side by side.
• This is key to get settings not located inside a settings/data JSON file.