The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
[x] Add the new Zenodo archive DOI values to pudl/workspace/datastore.py.
[x] Run the datastore script to download the new year of data.
[x] Add the new year to the years in etl_full.yml and etl_fast.yml.
[x] Add the new year to the working_partitions in pudl/metadata/sources.py
[x] Update the information in pudl/package_data:
[x] file maps (probably the same)
[x] column maps (probably the same)
[x] page maps (probably the same)
[x] skip footer (probably the same)
[x] skip rows (probably the same)
[x] Launch dagit and refresh the code location.
[x] Materialize the raw_eia860 asset group. Look out for warnings in the logs about missing or extra columns. If they appear, check and update the package_data accordingly.
[x] Materialize the clean_eia860 asset group. You'll probably see some errors related to encoding. Take a look at which column it's talking about and look at metadata/resources/eia.py to see which encoder in CODE_METADATA to tweak.
[x] Update the validation test test_minmax_rows in test/validate/eia_test.py.
[ ] Run tox and troubleshoot what else might be broken! Might include things like:
e-belfer
commented
10 months ago
pudl/workspace/datastore.py.etl_full.ymlandetl_fast.yml.working_partitionsinpudl/metadata/sources.pypudl/package_data:raw_eia860asset group. Look out for warnings in the logs about missing or extra columns. If they appear, check and update thepackage_dataaccordingly.clean_eia860asset group. You'll probably see some errors related to encoding. Take a look at which column it's talking about and look atmetadata/resources/eia.pyto see which encoder inCODE_METADATAto tweak.test_minmax_rowsintest/validate/eia_test.py.toxand troubleshoot what else might be broken! Might include things like: