Hi - please see the following code and log output. The scheduled workflow does not seem to recover from a database connection outage. The below log shows the application running, then I manually restart the database. This causes the various stacktraces in the log. At the very end of the log file the application recovered, but the scheduled workflows did not.
This is latest DBOS Python 0.11 version and Postgres in a container.
My expectation was that DBOS recovers completely, including scheduled workflows, surviving a database outage without any external recovery mechanism (that would have to be recoverable as well).
Do you have any suggestion how to handle this case? Did I miss any parameter or configuration setting to enable it?
Thank you.
import sys
import time
import traceback
from datetime import datetime
from dbos import DBOS
DBOS()
@DBOS.scheduled("*/5 * * * * *")
@DBOS.workflow()
def print_log(scheduled_time: datetime,
actual_time: datetime) -> None:
DBOS.logger.info("Workflow ran")
def main(arguments: list[str]) -> None:
DBOS().launch()
while True:
time.sleep(5)
DBOS.logger.info('alive')
if __name__ == "__main__":
try:
main(sys.argv)
except KeyboardInterrupt:
DBOS.logger.warning('KeyboardInterrupt received')
except Exception as e:
DBOS.logger.error(traceback.format_exc())
finally:
DBOS.destroy()
"C:\Users\miso\VirtualEnvironments\drag\Scripts\python.exe" "C:\Users\miso\Workspaces\drag\z_research\db_outage.py" "C:\Users\miso\Workspaces\drag\config"
15:58:21 [ INFO] (dbos:dbos.py:268) Initializing DBOS
15:58:23 [ INFO] (dbos:dbos.py:396) DBOS launched
15:58:25 [ INFO] (dbos:db_outage.py:15) Workflow ran
15:58:28 [ INFO] (dbos:db_outage.py:23) alive
15:58:30 [ INFO] (dbos:db_outage.py:15) Workflow ran
15:58:33 [ INFO] (dbos:db_outage.py:23) alive
15:58:35 [ INFO] (dbos:db_outage.py:15) Workflow ran
15:58:38 [ INFO] (dbos:db_outage.py:23) alive
15:58:40 [ INFO] (dbos:db_outage.py:15) Workflow ran
15:58:42 [ ERROR] (dbos:system_database.py:835) Notification listener error: consuming input failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
15:58:42 [ ERROR] (dbos:recovery.py:28) Exception encountered when recovering workflows: Traceback (most recent call last):
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 1967, in _exec_single_context
self.dialect.do_execute(
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 941, in do_execute
cursor.execute(statement, parameters)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\cursor.py", line 97, in execute
raise ex.with_traceback(None)
psycopg.OperationalError: consuming input failed: could not receive data from server: Software caused connection abort (0x00002745/10053)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\recovery.py", line 23, in _startup_recovery_thread
_execute_workflow_id(dbos, workflowID)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\core.py", line 235, in _execute_workflow_id
status = dbos._sys_db.get_workflow_status(workflow_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\system_database.py", line 356, in get_workflow_status
row = c.execute(
^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 1418, in execute
return meth(
^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\sql\elements.py", line 515, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 1640, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 1846, in _execute_context
return self._exec_single_context(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 1986, in _exec_single_context
self._handle_dbapi_exception(
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 2355, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 1967, in _exec_single_context
self.dialect.do_execute(
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 941, in do_execute
cursor.execute(statement, parameters)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\cursor.py", line 97, in execute
raise ex.with_traceback(None)
sqlalchemy.exc.OperationalError: (psycopg.OperationalError) consuming input failed: could not receive data from server: Software caused connection abort (0x00002745/10053)
[SQL: SELECT dbos.workflow_status.status, dbos.workflow_status.name, dbos.workflow_status.request, dbos.workflow_status.recovery_attempts, dbos.workflow_status.config_name, dbos.workflow_status.class_name, dbos.workflow_status.authenticated_user, dbos.workflow_status.authenticated_roles, dbos.workflow_status.assumed_role, dbos.workflow_status.queue_name
FROM dbos.workflow_status
WHERE dbos.workflow_status.workflow_uuid = %(workflow_uuid_1)s::VARCHAR]
[parameters: {'workflow_uuid_1': 'sched-run_every_minute-2024-09-29T14:07:00+00:00'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
15:58:42 [ WARNING] (dbos:queue.py:64) Exception encountered in queue thread: Traceback (most recent call last):
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
dbapi_connection = rec.get_connection()
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 866, in get_connection
self.__connect()
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
raise last_ex.with_traceback(None)
psycopg.OperationalError: connection failed: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\queue.py", line 60, in queue_thread
wf_ids = dbos._sys_db.start_queued_workflows(queue)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\system_database.py", line 1099, in start_queued_workflows
with self.engine.begin() as c:
File "C:\Installation\Python\Python312\Lib\contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3242, in begin
with self.connect() as conn:
^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3278, in connect
return self._connection_cls(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 148, in __init__
Connection._handle_dbapi_exception_noconnection(
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 2442, in _handle_dbapi_exception_noconnection
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
dbapi_connection = rec.get_connection()
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 866, in get_connection
self.__connect()
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
raise last_ex.with_traceback(None)
sqlalchemy.exc.OperationalError: (psycopg.OperationalError) connection failed: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
(Background on this error at: https://sqlalche.me/e/20/e3q8)
15:58:43 [ INFO] (dbos:db_outage.py:23) alive
15:58:43 [ ERROR] (dbos:system_database.py:835) Notification listener error: connection failed: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
15:58:43 [ WARNING] (dbos:queue.py:64) Exception encountered in queue thread: Traceback (most recent call last):
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
dbapi_connection = rec.get_connection()
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 837, in get_connection
self.__connect()
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
raise last_ex.with_traceback(None)
psycopg.OperationalError: connection failed: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\queue.py", line 60, in queue_thread
wf_ids = dbos._sys_db.start_queued_workflows(queue)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\system_database.py", line 1099, in start_queued_workflows
with self.engine.begin() as c:
File "C:\Installation\Python\Python312\Lib\contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3242, in begin
with self.connect() as conn:
^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3278, in connect
return self._connection_cls(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 148, in __init__
Connection._handle_dbapi_exception_noconnection(
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 2442, in _handle_dbapi_exception_noconnection
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
dbapi_connection = rec.get_connection()
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 837, in get_connection
self.__connect()
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
raise last_ex.with_traceback(None)
sqlalchemy.exc.OperationalError: (psycopg.OperationalError) connection failed: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
(Background on this error at: https://sqlalche.me/e/20/e3q8)
15:58:44 [ ERROR] (dbos:system_database.py:835) Notification listener error: connection failed: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
15:58:45 [ WARNING] (dbos:queue.py:64) Exception encountered in queue thread: Traceback (most recent call last):
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
dbapi_connection = rec.get_connection()
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 837, in get_connection
self.__connect()
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
raise last_ex.with_traceback(None)
psycopg.OperationalError: connection failed: connection to server at "127.0.0.1", port 5432 failed: FATAL: the database system is starting up
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\queue.py", line 60, in queue_thread
wf_ids = dbos._sys_db.start_queued_workflows(queue)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\system_database.py", line 1099, in start_queued_workflows
with self.engine.begin() as c:
File "C:\Installation\Python\Python312\Lib\contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3242, in begin
with self.connect() as conn:
^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3278, in connect
return self._connection_cls(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 148, in __init__
Connection._handle_dbapi_exception_noconnection(
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 2442, in _handle_dbapi_exception_noconnection
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
dbapi_connection = rec.get_connection()
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 837, in get_connection
self.__connect()
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
raise last_ex.with_traceback(None)
sqlalchemy.exc.OperationalError: (psycopg.OperationalError) connection failed: connection to server at "127.0.0.1", port 5432 failed: FATAL: the database system is starting up
(Background on this error at: https://sqlalche.me/e/20/e3q8)
Exception in thread Thread-5 (scheduler_loop):
Traceback (most recent call last):
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
dbapi_connection = rec.get_connection()
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 837, in get_connection
self.__connect()
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
raise last_ex.with_traceback(None)
psycopg.OperationalError: connection failed: connection to server at "127.0.0.1", port 5432 failed: FATAL: the database system is starting up
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Installation\Python\Python312\Lib\threading.py", line 1073, in _bootstrap_inner
self.run()
File "C:\Installation\Python\Python312\Lib\threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\scheduler\scheduler.py", line 34, in scheduler_loop
scheduler_queue.enqueue(func, nextExecTime, datetime.now(timezone.utc))
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\queue.py", line 51, in enqueue
return _start_workflow(dbos, func, self.name, False, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\core.py", line 410, in _start_workflow
status = _init_workflow(
^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\core.py", line 162, in _init_workflow
dbos._sys_db.update_workflow_status(
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\system_database.py", line 295, in update_workflow_status
with self.engine.begin() as c:
File "C:\Installation\Python\Python312\Lib\contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3242, in begin
with self.connect() as conn:
^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3278, in connect
return self._connection_cls(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 148, in __init__
Connection._handle_dbapi_exception_noconnection(
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 2442, in _handle_dbapi_exception_noconnection
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
dbapi_connection = rec.get_connection()
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 837, in get_connection
self.__connect()
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
with util.safe_reraise():
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
raise last_ex.with_traceback(None)
sqlalchemy.exc.OperationalError: (psycopg.OperationalError) connection failed: connection to server at "127.0.0.1", port 5432 failed: FATAL: the database system is starting up
(Background on this error at: https://sqlalche.me/e/20/e3q8)
15:58:48 [ INFO] (dbos:db_outage.py:23) alive
15:58:53 [ INFO] (dbos:db_outage.py:23) alive
15:58:58 [ INFO] (dbos:db_outage.py:23) alive
15:59:03 [ INFO] (dbos:db_outage.py:23) alive
15:59:08 [ INFO] (dbos:db_outage.py:23) alive
15:59:13 [ INFO] (dbos:db_outage.py:23) alive
15:59:18 [ INFO] (dbos:db_outage.py:23) alive
15:59:23 [ INFO] (dbos:db_outage.py:23) alive
15:59:28 [ WARNING] (dbos:db_outage.py:30) KeyboardInterrupt received
Process finished with exit code -1
Hi - please see the following code and log output. The scheduled workflow does not seem to recover from a database connection outage. The below log shows the application running, then I manually restart the database. This causes the various stacktraces in the log. At the very end of the log file the application recovered, but the scheduled workflows did not.
This is latest DBOS Python 0.11 version and Postgres in a container.
My expectation was that DBOS recovers completely, including scheduled workflows, surviving a database outage without any external recovery mechanism (that would have to be recoverable as well).
Do you have any suggestion how to handle this case? Did I miss any parameter or configuration setting to enable it?
Thank you.