Research and Document Elastic Map Reduce sizing, cost of operation and other dev ops parameters

[PeterCiuffetti]

I need to do a little more research and documentation on managing EMR clusters from a dev ops perspective.

Among the things I do not know how to do:

• Backup and restore the data on the Hadoop file system
• Resize a cluster
• Resize the hadoop file system
• Whether there are any auto-scaling features
• Best practices for deploying code to a running cluster (or more graceful mechanisms for stopping it at a convenient juncture of its continuous loop, updating its code, and restarting it)
• Delete URLs from the Nutch database (I've done some of this ssh'ed into the cluster, but I want to be able to do this remotely)
• Add URLs to a crawl while it is running
• Knowing when we've reached the 'bottom' of any site.
• Adjusting the crawl schedule on a per-site basis

I should also study the cost of operation a little more closely so we can budget the cost of CoherentBot on a cost-per-net PDF harvested. Then if a new size with x thousand documents is added to the list, we can estimate what it costs on a per-site basis.

There are many other knobs an levers we should consider being required down the road

• Including a site who's results are exported to another product (this can probably be a collection attribute that CB is not too concerned with itself)
• Eventually supporting HTML harvesting, particularly POCO-like, CMS-driven sites with 'view' pages we want to export as artifacts.
• Handling other crawling agendas, like crawling for person names. This can be independent clusters with config files quite different than PDF crawlers
• Using content from the Wayback Machine (again, probably another cluster designed just for this purpose)

[PeterCiuffetti]

Marking this as complexity M ... its probably 2 or 3 days of research, setup, trial and error. And I also want to document what I learn so others can take on CoherenceBot dev ops tasks.

[PeterCiuffetti]

See Google Doc "CoherenceBot Dev Ops" https://docs.google.com/document/d/14eQ80aA-m5x30dUeh2hiJWx29eZeBZpJVGloV_9k0As/edit#