cfpb / cfgov-crawler-app

An electron app which crawls consumerfinance.gov and gathers interesting data
6 stars 8 forks source link

Add ability to run crawl from command line #6

Open chosak opened 2 years ago

chosak commented 2 years ago

It would be nice to be able to start a crawl from the command line, without needing to interact with a UI. This will allow use of this project as part of regularly scheduled automation.

A nice-to-have would be to have text-based console output like the graphical interface, to give an idea how far along we are in the crawl. An even nicer-to-have would be the ability to pause/resume the crawl as we can do in the graphical UI.

Presumably there's some nice Node CLI package we could leverage to support the above? Ping @mistergone @contolini.