search

dbos-inc / dbos-transact-py

Ultra-Lightweight Durable Execution in Python
https://docs.dbos.dev
MIT License
244 stars 7 forks source link

Scheduled workflow does not recover from database connection outage? #139

Open chbussler opened 4 days ago

chbussler commented 4 days ago

Hi - please see the following code and log output. The scheduled workflow does not seem to recover from a database connection outage. The below log shows the application running, then I manually restart the database. This causes the various stacktraces in the log. At the very end of the log file the application recovered, but the scheduled workflows did not.

This is latest DBOS Python 0.11 version and Postgres in a container.

My expectation was that DBOS recovers completely, including scheduled workflows, surviving a database outage without any external recovery mechanism (that would have to be recoverable as well).

Do you have any suggestion how to handle this case? Did I miss any parameter or configuration setting to enable it?

Thank you.

import sys
import time
import traceback
from datetime import datetime

from dbos import DBOS

DBOS()

@DBOS.scheduled("*/5 * * * * *")
@DBOS.workflow()
def print_log(scheduled_time: datetime,
              actual_time: datetime) -> None:
    DBOS.logger.info("Workflow ran")

def main(arguments: list[str]) -> None:
    DBOS().launch()

    while True:
        time.sleep(5)
        DBOS.logger.info('alive')

if __name__ == "__main__":
    try:
        main(sys.argv)
    except KeyboardInterrupt:
        DBOS.logger.warning('KeyboardInterrupt received')
    except Exception as e:
        DBOS.logger.error(traceback.format_exc())
    finally:
        DBOS.destroy()

"C:\Users\miso\VirtualEnvironments\drag\Scripts\python.exe" "C:\Users\miso\Workspaces\drag\z_research\db_outage.py" "C:\Users\miso\Workspaces\drag\config" 
15:58:21 [    INFO] (dbos:dbos.py:268) Initializing DBOS
15:58:23 [    INFO] (dbos:dbos.py:396) DBOS launched
15:58:25 [    INFO] (dbos:db_outage.py:15) Workflow ran
15:58:28 [    INFO] (dbos:db_outage.py:23) alive
15:58:30 [    INFO] (dbos:db_outage.py:15) Workflow ran
15:58:33 [    INFO] (dbos:db_outage.py:23) alive
15:58:35 [    INFO] (dbos:db_outage.py:15) Workflow ran
15:58:38 [    INFO] (dbos:db_outage.py:23) alive
15:58:40 [    INFO] (dbos:db_outage.py:15) Workflow ran
15:58:42 [   ERROR] (dbos:system_database.py:835) Notification listener error: consuming input failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
15:58:42 [   ERROR] (dbos:recovery.py:28) Exception encountered when recovering workflows: Traceback (most recent call last):
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 941, in do_execute
    cursor.execute(statement, parameters)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\cursor.py", line 97, in execute
    raise ex.with_traceback(None)
psycopg.OperationalError: consuming input failed: could not receive data from server: Software caused connection abort (0x00002745/10053)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\recovery.py", line 23, in _startup_recovery_thread
    _execute_workflow_id(dbos, workflowID)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\core.py", line 235, in _execute_workflow_id
    status = dbos._sys_db.get_workflow_status(workflow_id)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\system_database.py", line 356, in get_workflow_status
    row = c.execute(
          ^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 1418, in execute
    return meth(
           ^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\sql\elements.py", line 515, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 1640, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 2355, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 941, in do_execute
    cursor.execute(statement, parameters)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\cursor.py", line 97, in execute
    raise ex.with_traceback(None)
sqlalchemy.exc.OperationalError: (psycopg.OperationalError) consuming input failed: could not receive data from server: Software caused connection abort (0x00002745/10053)
[SQL: SELECT dbos.workflow_status.status, dbos.workflow_status.name, dbos.workflow_status.request, dbos.workflow_status.recovery_attempts, dbos.workflow_status.config_name, dbos.workflow_status.class_name, dbos.workflow_status.authenticated_user, dbos.workflow_status.authenticated_roles, dbos.workflow_status.assumed_role, dbos.workflow_status.queue_name 
FROM dbos.workflow_status 
WHERE dbos.workflow_status.workflow_uuid = %(workflow_uuid_1)s::VARCHAR]
[parameters: {'workflow_uuid_1': 'sched-run_every_minute-2024-09-29T14:07:00+00:00'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

15:58:42 [ WARNING] (dbos:queue.py:64) Exception encountered in queue thread: Traceback (most recent call last):
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
    dbapi_connection = rec.get_connection()
                       ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 866, in get_connection
    self.__connect()
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
    return dialect.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
    raise last_ex.with_traceback(None)
psycopg.OperationalError: connection failed: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\queue.py", line 60, in queue_thread
    wf_ids = dbos._sys_db.start_queued_workflows(queue)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\system_database.py", line 1099, in start_queued_workflows
    with self.engine.begin() as c:
  File "C:\Installation\Python\Python312\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3242, in begin
    with self.connect() as conn:
         ^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3278, in connect
    return self._connection_cls(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 148, in __init__
    Connection._handle_dbapi_exception_noconnection(
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 2442, in _handle_dbapi_exception_noconnection
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
    dbapi_connection = rec.get_connection()
                       ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 866, in get_connection
    self.__connect()
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
    return dialect.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
    raise last_ex.with_traceback(None)
sqlalchemy.exc.OperationalError: (psycopg.OperationalError) connection failed: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
(Background on this error at: https://sqlalche.me/e/20/e3q8)

15:58:43 [    INFO] (dbos:db_outage.py:23) alive
15:58:43 [   ERROR] (dbos:system_database.py:835) Notification listener error: connection failed: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
15:58:43 [ WARNING] (dbos:queue.py:64) Exception encountered in queue thread: Traceback (most recent call last):
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
    dbapi_connection = rec.get_connection()
                       ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 837, in get_connection
    self.__connect()
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
    return dialect.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
    raise last_ex.with_traceback(None)
psycopg.OperationalError: connection failed: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\queue.py", line 60, in queue_thread
    wf_ids = dbos._sys_db.start_queued_workflows(queue)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\system_database.py", line 1099, in start_queued_workflows
    with self.engine.begin() as c:
  File "C:\Installation\Python\Python312\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3242, in begin
    with self.connect() as conn:
         ^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3278, in connect
    return self._connection_cls(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 148, in __init__
    Connection._handle_dbapi_exception_noconnection(
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 2442, in _handle_dbapi_exception_noconnection
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
    dbapi_connection = rec.get_connection()
                       ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 837, in get_connection
    self.__connect()
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
    return dialect.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
    raise last_ex.with_traceback(None)
sqlalchemy.exc.OperationalError: (psycopg.OperationalError) connection failed: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
(Background on this error at: https://sqlalche.me/e/20/e3q8)

15:58:44 [   ERROR] (dbos:system_database.py:835) Notification listener error: connection failed: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
15:58:45 [ WARNING] (dbos:queue.py:64) Exception encountered in queue thread: Traceback (most recent call last):
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
    dbapi_connection = rec.get_connection()
                       ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 837, in get_connection
    self.__connect()
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
    return dialect.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
    raise last_ex.with_traceback(None)
psycopg.OperationalError: connection failed: connection to server at "127.0.0.1", port 5432 failed: FATAL:  the database system is starting up

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\queue.py", line 60, in queue_thread
    wf_ids = dbos._sys_db.start_queued_workflows(queue)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\system_database.py", line 1099, in start_queued_workflows
    with self.engine.begin() as c:
  File "C:\Installation\Python\Python312\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3242, in begin
    with self.connect() as conn:
         ^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3278, in connect
    return self._connection_cls(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 148, in __init__
    Connection._handle_dbapi_exception_noconnection(
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 2442, in _handle_dbapi_exception_noconnection
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
    dbapi_connection = rec.get_connection()
                       ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 837, in get_connection
    self.__connect()
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
    return dialect.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
    raise last_ex.with_traceback(None)
sqlalchemy.exc.OperationalError: (psycopg.OperationalError) connection failed: connection to server at "127.0.0.1", port 5432 failed: FATAL:  the database system is starting up
(Background on this error at: https://sqlalche.me/e/20/e3q8)

Exception in thread Thread-5 (scheduler_loop):
Traceback (most recent call last):
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
    dbapi_connection = rec.get_connection()
                       ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 837, in get_connection
    self.__connect()
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
    return dialect.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
    raise last_ex.with_traceback(None)
psycopg.OperationalError: connection failed: connection to server at "127.0.0.1", port 5432 failed: FATAL:  the database system is starting up

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Installation\Python\Python312\Lib\threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "C:\Installation\Python\Python312\Lib\threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\scheduler\scheduler.py", line 34, in scheduler_loop
    scheduler_queue.enqueue(func, nextExecTime, datetime.now(timezone.utc))
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\queue.py", line 51, in enqueue
    return _start_workflow(dbos, func, self.name, False, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\core.py", line 410, in _start_workflow
    status = _init_workflow(
             ^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\core.py", line 162, in _init_workflow
    dbos._sys_db.update_workflow_status(
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\dbos\system_database.py", line 295, in update_workflow_status
    with self.engine.begin() as c:
  File "C:\Installation\Python\Python312\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3242, in begin
    with self.connect() as conn:
         ^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3278, in connect
    return self._connection_cls(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 148, in __init__
    Connection._handle_dbapi_exception_noconnection(
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 2442, in _handle_dbapi_exception_noconnection
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\base.py", line 3302, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 717, in checkout
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 715, in checkout
    dbapi_connection = rec.get_connection()
                       ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 837, in get_connection
    self.__connect()
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 900, in __connect
    with util.safe_reraise():
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\util\langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\pool\base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\create.py", line 643, in connect
    return dialect.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\sqlalchemy\engine\default.py", line 621, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\miso\VirtualEnvironments\drag\Lib\site-packages\psycopg\connection.py", line 119, in connect
    raise last_ex.with_traceback(None)
sqlalchemy.exc.OperationalError: (psycopg.OperationalError) connection failed: connection to server at "127.0.0.1", port 5432 failed: FATAL:  the database system is starting up
(Background on this error at: https://sqlalche.me/e/20/e3q8)
15:58:48 [    INFO] (dbos:db_outage.py:23) alive
15:58:53 [    INFO] (dbos:db_outage.py:23) alive
15:58:58 [    INFO] (dbos:db_outage.py:23) alive
15:59:03 [    INFO] (dbos:db_outage.py:23) alive
15:59:08 [    INFO] (dbos:db_outage.py:23) alive
15:59:13 [    INFO] (dbos:db_outage.py:23) alive
15:59:18 [    INFO] (dbos:db_outage.py:23) alive
15:59:23 [    INFO] (dbos:db_outage.py:23) alive
15:59:28 [ WARNING] (dbos:db_outage.py:30) KeyboardInterrupt received

Process finished with exit code -1
qianl15 commented 18 hours ago

Thanks for reporting this issue! Looks like a bug. We're looking into this.