Scrapy job persistence

[jpmckinney]

This is to allow jobs to be paused/resumed, e.g. when re-deploying Kingfisher Collect as a whole to install new requirements.

Scrapy can pause/resume specific crawls https://docs.scrapy.org/en/latest/topics/jobs.html but it's not clear how to configure it to do that for all crawls by default https://github.com/scrapy/scrapy/issues/3416

Can also look into how the following maintain state:

• scrapy/scrapyd#359
• https://github.com/my8100/scrapydweb
• my8100/scrapydweb#21
• https://github.com/my8100/scrapydweb/blob/2df1c3d5dcf92dd1a8b11eb5cf52663afb751316/scrapydweb/views/dashboard/jobs.py

[jpmckinney]

Split out of #79

[jpmckinney]

The next version of Scrapyd will have a job persistence option! https://github.com/scrapy/scrapyd/pull/418

[jpmckinney]

Might just require a one-line configuration: https://scrapyd.readthedocs.io/en/stable/config.html#jobstorage