refactor of how the crawl is called via commandline (puppeteer and cheerio)
-- flags, n indicate number of maxrequests, different test databases, batchfilescope and fullscope
includes being able to download a csv from the frontend
includes the csv to download failed links
email when the crawler stops
a forever script to help restart the process if it stops (needs to be implemented)
this merge includes the following: