The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
[ ] Add the new Zenodo archive DOI values to pudl/workspace/datastore.py.
[ ] Run the datastore script to download the new data: pudl_datastore --dataset eia923. The new raw data will appear in pudl_input/eia923/<ZENODO_DOI>/...
[ ] Update the information in pudl/package_data/eia923 if necessary:
[ ] file maps
[ ] column maps
[ ] page maps
[ ] skip footer
[ ] skip rows
[ ] Launch dagit and refresh the code location (run in your terminaldagster-webserver -m pudl.etl and then open http://127.0.0.1:3000/locations/pudl.etl/jobs/etl_full in a browser)
[ ] Materialize the raw_eia923 asset group. Look out for warnings in the logs about missing or extra columns. If they appear, check and update the package_data accordingly.
[ ] Materialize the _core_eia923 asset group. Look out for warnings and fix accordingly.
[ ] Materialize the norm_eia and then denorm_eia asset groups. You'll may see some errors related to encoding. Take a look at which column it's talking about and look at metadata/resources/eia.py to see which encoder in CODE_METADATA to tweak.
Annual Updates Docs: https://catalystcoop-pudl.readthedocs.io/en/dev/dev/annual_updates.html
pudl/workspace/datastore.py
.pudl_datastore --dataset eia923
. The new raw data will appear inpudl_input/eia923/<ZENODO_DOI>/...
pudl/package_data/eia923
if necessary:dagster-webserver -m pudl.etl
and then openhttp://127.0.0.1:3000/locations/pudl.etl/jobs/etl_full
in a browser)raw_eia923
asset group. Look out for warnings in the logs about missing or extra columns. If they appear, check and update thepackage_data
accordingly._core_eia923
asset group. Look out for warnings and fix accordingly.norm_eia
and thendenorm_eia
asset groups. You'll may see some errors related to encoding. Take a look at which column it's talking about and look atmetadata/resources/eia.py
to see which encoder inCODE_METADATA
to tweak.make pytest-integration-full
in the terminalmake pytest-validation
in the terminal