Closed ezwelty closed 3 years ago
In all of these cases we're looking for a path to a human readable "source" of the data right -- not the raw downloads. Both FERC and MSHA have overhauled their sites in the last year, and the old URLs no longer work, even though the raw data download location is the same in many cases. I would say...
msha
: https://arlweb.msha.gov/OpenGovernmentData/OGIMSHA.aspferc1
: https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-1-electric-utility-annualferceqr
: https://www.ferc.gov/industries-data/electric/power-sales-and-markets/electric-quarterly-reports-eqrferc714
: https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-no-714-annual-electricepacems
: https://ampd.epa.gov/ampdeia861
: https://www.eia.gov/electricity/data/eia861@zaneselvans Thanks for the quick response. I agree that human readable is the right choice. I'll implement your choices.
i believe this is implemented in #806
When attempting to merge source attributes spread out across
src/pudl/package_data/meta/datapackage.json
,pudl.constants.data_source_info
andpudl.constants.base_data_urls
, I found some differences in thepath
attribute. I can't find any reason why they would differ.Unless there is a good reason to have multiple paths for a source, we need to choose one
path
from the options for each source below:path
(datapackage.json
)path
(constants.data_source_info
)mhsa
ferceqr
ferc1
epacems
Note that this is after fixing some of the
path
insrc/pudl/package_data/meta/datapackage.json
as follows:eia861
: https://www.eia.gov/electricity/data/eia861/zip (forbidden) to https://www.eia.gov/electricity/data/eia861msha
: https://arlweb.msha.gov/OpenGovernmentData/DataSets (forbidden) to https://arlweb.msha.gov/OpenGovernmentData/OGIMSHA.aspepaipm
: https://www.epa.gov/sites/production/files/2019-03 (not found) to https://www.epa.gov/sites/production/files/2019-03/documents/chapter_1.pdfferc714
: https://www.ferc.gov/docs-filing/forms/form-714/data (not found) to https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-no-714-annual-electric