The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
When trying to do a fresh run of ferc_to_sqlite_full I'm getting the error below. It looks like this is because there's only data from 2021 and 2022 at https://zenodo.org/record/8006881, which is where the data is (attempting to) being fetch from.
Bug Severity
How badly is this bug affecting you?
High: This bug is preventing me from using PUDL.
To Reproduce
Run xbrl2sqliteop with the full config.
dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "xbrl2sqlite":
File "/Users/dstansby/mambaforge/envs/pudl/lib/python3.11/site-packages/dagster/_core/execution/plan/execute_plan.py", line 273, in dagster_event_sequence_for_step
for step_event in check.generator(step_events):
File "/Users/dstansby/mambaforge/envs/pudl/lib/python3.11/site-packages/dagster/_core/execution/plan/execute_step.py", line 369, in core_dagster_event_sequence_for_step
for user_event in check.generator(
File "/Users/dstansby/mambaforge/envs/pudl/lib/python3.11/site-packages/dagster/_core/execution/plan/execute_step.py", line 90, in _step_output_error_checked_user_event_sequence
for user_event in user_event_sequence:
File "/Users/dstansby/mambaforge/envs/pudl/lib/python3.11/site-packages/dagster/_core/execution/plan/compute.py", line 192, in execute_core_compute
for step_output in _yield_compute_results(step_context, inputs, compute_fn):
File "/Users/dstansby/mambaforge/envs/pudl/lib/python3.11/site-packages/dagster/_core/execution/plan/compute.py", line 161, in _yield_compute_results
for event in iterate_with_context(
File "/Users/dstansby/mambaforge/envs/pudl/lib/python3.11/site-packages/dagster/_utils/__init__.py", line 443, in iterate_with_context
with context_fn():
File "/Users/dstansby/mambaforge/envs/pudl/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/Users/dstansby/mambaforge/envs/pudl/lib/python3.11/site-packages/dagster/_core/execution/plan/utils.py", line 84, in op_execution_error_boundary
raise error_cls(
The above exception was caused by the following exception:
KeyError: "No resources found for ferc2: {'year': 1996, 'data_format': 'xbrl'}"
File "/Users/dstansby/mambaforge/envs/pudl/lib/python3.11/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
yield
File "/Users/dstansby/mambaforge/envs/pudl/lib/python3.11/site-packages/dagster/_utils/__init__.py", line 445, in iterate_with_context
next_output = next(iterator)
^^^^^^^^^^^^^^
File "/Users/dstansby/mambaforge/envs/pudl/lib/python3.11/site-packages/dagster/_core/execution/plan/compute_generator.py", line 124, in _coerce_op_compute_fn_to_iterator
result = invoke_compute_fn(
^^^^^^^^^^^^^^^^^^
File "/Users/dstansby/mambaforge/envs/pudl/lib/python3.11/site-packages/dagster/_core/execution/plan/compute_generator.py", line 118, in invoke_compute_fn
return fn(context, **args_to_pass) if context_arg_provided else fn(**args_to_pass)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dstansby/github/pudl/src/pudl/extract/xbrl.py", line 151, in xbrl2sqlite
convert_form(
File "/Users/dstansby/github/pudl/src/pudl/extract/xbrl.py", line 190, in convert_form
raw_archive, taxonomy_entry_point = datastore.get_taxonomy(year, form)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dstansby/github/pudl/src/pudl/extract/xbrl.py", line 29, in get_taxonomy
raw_archive = self.datastore.get_unique_resource(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dstansby/github/pudl/src/pudl/workspace/datastore.py", line 405, in get_unique_resource
raise KeyError(f"No resources found for {dataset}: {filters}")
The above exception occurred during handling of the following exception:
StopIteration
File "/Users/dstansby/github/pudl/src/pudl/workspace/datastore.py", line 403, in get_unique_resource
_, content = next(res)
^^^^^^^^^
Expected behavior
Op runs without any errors
Software Environment?
Operating System. MacOS 15
Python version and distribution Python 3.11.4
How did you install PUDL? Using git, on dev branch
Describe the bug
When trying to do a fresh run of
ferc_to_sqlite_full
I'm getting the error below. It looks like this is because there's only data from 2021 and 2022 at https://zenodo.org/record/8006881, which is where the data is (attempting to) being fetch from.Bug Severity
How badly is this bug affecting you?
To Reproduce
Run
xbrl2sqlite
op with the full config.Expected behavior
Op runs without any errors
Software Environment?
Additional context
Add any other context about the problem here.