The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
[x] Add the new Zenodo archive DOI values to pudl/workspace/datastore.py.
[x] Run the datastore script to download the new year of data.
[x] Add the new year to the years in etl_full.yml and etl_fast.yml.
[x] Add the new year to the working_partitions in pudl/metadata/sources.py (note that eia860 is linked to eia860m and you must also change the working_partition in eia860m to the new year).
[x] Update the information in pudl/package_data:
[x] file maps (add _Early_Release suffix!)
[x] column maps (probably the same)
[x] page maps (probably the same)
[x] skip footer (probably the same)
[x] skip rows (+1 for early release data)
[x] The EIA 860 is linked to the EIA860m. For now, turn off the checks there by setting eia860m: bool = False in settings.pyandetl_fast.py
[x] Launch dagit and refresh the code location.
[x] Materialize the raw_eia860 asset group. Look out for warnings in the logs about missing or extra columns. If they appear, check and update the package_data accordingly.
[x] Materialize the clean_eia923 asset group. You'll probably see some errors related to encoding. Take a look at which column it's talking about and look at metadata/resources/eia.py to see which encoder in CODE_METADATA to tweak.
[x] Update the validation test test_minmax_rows in test/validate/eia_test.py.
e-belfer
commented
1 year ago
pudl/workspace/datastore.py.etl_full.ymlandetl_fast.yml.working_partitionsinpudl/metadata/sources.py(note that eia860 is linked to eia860m and you must also change theworking_partitionin eia860m to the new year).pudl/package_data:eia860m: bool = Falseinsettings.pyandetl_fast.pymetadata/resources/eia.pyto see which encoder inCODE_METADATAto tweak.test_minmax_rowsintest/validate/eia_test.py.