...due to non-existing table demand.egon_etrago_electricity_cts.
This table seems to be created in Task electricity_demand.temporal.insert-cts-loadhere.
The task depends on cts_electricity_demand_annual which is CtsElectricityDemand (see pipeline.py) which includes the insert-cts-load task. This in turn depends on demandregio, so I expect the original task to fail due to a failing upstream task and thus not to be executed. But it is.
Maybe it's just a dependency problem I don't see and it never rose since the electricity demand has always been present when the task started...
Did anyone stumble across this problem? Maybe you @ClaraBuettner ?
Full log:
[2024-01-10 11:13:47,854] {standard_task_runner.py:78} INFO - Job 3: Subtask DSM_cts_ind.dsm-cts-ind-processing
[2024-01-10 11:13:47,951] {logging_mixin.py:120} INFO - Running <TaskInstance: egon-data-processing-pipeline.DSM_cts_ind.dsm-cts-ind-processing 2024-01-10T10:13:36.809164+00:00 [running]> on host ws-02lin.rl-institut.local
[2024-01-10 11:13:48,048] {logging_mixin.py:120} INFO -
[2024-01-10 11:13:48,048] {logging_mixin.py:120} INFO - CTS per osm-area: cooling, ventilation and air conditioning
[2024-01-10 11:13:48,048] {logging_mixin.py:120} INFO -
[2024-01-10 11:13:48,191] {taskinstance.py:1150} ERROR - (psycopg2.errors.UndefinedTable) relation "demand.egon_etrago_electricity_cts" does not exist
LINE 2: demand.egon_etrago_electricity_cts
^
[SQL: SELECT bus_id, scn_name, p_set FROM
demand.egon_etrago_electricity_cts]
(Background on this error at: http://sqlalche.me/e/13/f405)
Traceback (most recent call last):
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "demand.egon_etrago_electricity_cts" does not exist
LINE 2: demand.egon_etrago_electricity_cts
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/code/src/egon/data/datasets/__init__.py", line 202, in skip_task
result = super(type(task), task).execute(*xs, **ks)
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/airflow/operators/python_operator.py", line 113, in execute
return_value = self.execute_callable()
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/code/src/egon/data/datasets/DSM_cts_ind.py", line 1621, in dsm_cts_ind_processing
dsm_cts_ind()
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/code/src/egon/data/datasets/DSM_cts_ind.py", line 1093, in dsm_cts_ind
dsm = cts_data_import(cts_cool_vent_ac_share)
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/code/src/egon/data/datasets/DSM_cts_ind.py", line 224, in cts_data_import
ts = db.select_dataframe(
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/code/src/egon/data/db.py", line 188, in select_dataframe
df = pd.read_sql(sql, engine(), index_col=index_col)
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/pandas/io/sql.py", line 628, in read_sql
return pandas_sql.read_query(
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/pandas/io/sql.py", line 1579, in read_query
result = self.execute(*args)
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/pandas/io/sql.py", line 1424, in execute
return self.connectable.execution_options().execute(*args, **kwargs)
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2235, in execute
return connection.execute(statement, *multiparams, **params)
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1003, in execute
return self._execute_text(object_, multiparams, params)
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1172, in _execute_text
ret = self._execute_context(
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
self._handle_dbapi_exception(
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
util.raise_(
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "demand.egon_etrago_electricity_cts" does not exist
LINE 2: demand.egon_etrago_electricity_cts
^
[SQL: SELECT bus_id, scn_name, p_set FROM
demand.egon_etrago_electricity_cts]
(Background on this error at: http://sqlalche.me/e/13/f405)
[2024-01-10 11:13:48,193] {taskinstance.py:1187} INFO - Marking task as FAILED. dag_id=egon-data-processing-pipeline, task_id=DSM_cts_ind.dsm-cts-ind-processing, execution_date=20240110T101336, start_date=20240110T101347, end_date=20240110T101348
[2024-01-10 11:13:52,682] {local_task_job.py:102} INFO - Task exited with return code 1
...due to non-existing table
demand.egon_etrago_electricity_cts. This table seems to be created in Taskelectricity_demand.temporal.insert-cts-loadhere.The task depends on
cts_electricity_demand_annualwhich isCtsElectricityDemand(see pipeline.py) which includes theinsert-cts-loadtask. This in turn depends on demandregio, so I expect the original task to fail due to a failing upstream task and thus not to be executed. But it is.Maybe it's just a dependency problem I don't see and it never rose since the electricity demand has always been present when the task started...
Did anyone stumble across this problem? Maybe you @ClaraBuettner ?
Full log: