Try repeated scraping / uploading

[ptvirgo]

Make sure you can run the scraper & zenodo uploader from https://github.com/catalyst-cooperative/infrastructure, and that it works consistently.

[zaneselvans]

Usage Issues:

• Default behavior isn’t specified in the --help message for epacems.py. E.g. it’s unclear what happens if one doesn’t specify a state, or a month. (From running the script, it appears that the default behavior is to get all of them).
• Can lists of states / years / months be specified on the command line? Or just one at a time?
• Shouldn’t the package be PudlScrapers, not PudleScrapers? Does the InterCapping conform to Python package naming conventions?
• What do we think about using different python environment management systems in different repositories going forward? (though this package is meant to be installed locally from the repo, using pip, so it can be installed into an existing conda environment wherever.)
• Is there any reason not to integrate the packages in the requirements.txt directly into setup.py via the install_requires parameter?
• The use of the name pudl for this module collides with our main pudl module if you install them in the same environment. I installed the scrapers module in my pudl-dev environment and then everything about pudl broke.
• Would be nice to have a checklist / guide for what needs to be done in general to add another data source to the existing collection? (E.g. for FERC 714) in the README. Maybe there could be a dummy template dataset with comments explaining what all the parts need to contain?

Style / Code:

• In general we're not trying to be compatible with Python 2 at all, and other PUDL tools require Python 3.7 at least, so we can use f-string formatting anywhere we want.
• We're also trying to move to using pathlib everywhere instead of os.path
• Would be nice to have more comments in everything so that if one of the other of us need to go in and maintain things later, it's clear what the various parts are doing, with parsing the pages, etc.
• Does the Cems object in items actually get used? Or was that all offloaded into the CEMS script? We've been calling it epacems everywhere else so it should probably be EpaCems for consistency.

Structural Questions:

• The scrapy.cfg refers to pudl.settings but where is that set? Is it just the settings in the pudl/settings.py module? How does it detect what things are available to be scraped (i.e. the things listed in scrapy list?)
• What are the high level roles of the "items", "middlewares", and "pipelines"? How do they relate to each other and the data source specific spiders?