search

PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.46k stars 1.52k forks source link

Flaky test `test_service_sends_many_notifications_and_clears_queue` #9848

Closed zanieb closed 1 year ago

zanieb commented 1 year ago

First check

Description

Test timed out with

----------------------------- Captured stderr call -----------------------------

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

~~~~~~~~~~~~~~~~~~~~ Stack of Thread-3676 (139955636524608) ~~~~~~~~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/core.py", line 99, in run
    future, function = self._tx.get(timeout=0.1)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/queue.py", line 180, in get
    self.not_empty.wait(remaining)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 324, in wait
    gotit = waiter.acquire(True, timeout)

~~~~~~~~~~~~~~~~~~~~~ Stack of <unknown> (139955670087232) ~~~~~~~~~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 1042, in _bootstrap_inner
    self._delete()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/core.py", line 103, in run
    break
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/queue.py", line 179, in get
    raise Empty
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 332, in wait
    self._waiters.remove(waiter)

~~~~~~~~~~~~~~~~ Stack of EventsWorkerThread (139955653305920) ~~~~~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/home/runner/work/prefect/prefect/src/prefect/_internal/concurrency/threads.py", line 90, in _entrypoint
    self._run_until_shutdown()
  File "/home/runner/work/prefect/prefect/src/prefect/_internal/concurrency/threads.py", line 105, in _run_until_shutdown
    task = call.run()
  File "/home/runner/work/prefect/prefect/src/prefect/_internal/concurrency/calls.py", line 249, in run
    coro = self.context.run(self._run_sync)
  File "/home/runner/work/prefect/prefect/src/prefect/_internal/concurrency/calls.py", line 316, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/queue.py", line 171, in get
    self.not_empty.wait()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 320, in wait
    waiter.acquire()

~~~~~~~~~~~~~~~~ Stack of AnyIO worker thread (139955804304960) ~~~~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 797, in run
    item = self.queue.get()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/queue.py", line 171, in get
    self.not_empty.wait()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 320, in wait
    waiter.acquire()

~~~~~~~~~~~~~~~ Stack of GlobalEventLoopThread (139955535869504) ~~~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/home/runner/work/prefect/prefect/src/prefect/_internal/concurrency/threads.py", line 198, in _entrypoint
    asyncio.run(self._run_until_shutdown())
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete
    self.run_forever()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/asyncio/base_events.py", line 607, in run_forever
    self._run_once()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/asyncio/base_events.py", line 1884, in _run_once
    event_list = self._selector.select(timeout)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)

~~~~~~~~~~~~~~~~~~~~~ Stack of asyncio_0 (139955770742336) ~~~~~~~~~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/concurrent/futures/thread.py", line 81, in _worker
    work_item = work_queue.get(block=True)

~~~~~~~~~~~~~~~~~~~~~ Stack of waitpid-0 (139955787523648) ~~~~~~~~~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/asyncio/unix_events.py", line 1403, in _do_waitpid
    pid, status = os.waitpid(expected_pid, 0)

~~~~~~~~~~~~~~~~~~~~~ Stack of <unknown> (139957129008704) ~~~~~~~~~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/execnet/gateway_base.py", line 285, in _perform_spawn
    reply.run()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/execnet/gateway_base.py", line 220, in run
    self._result = func(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/execnet/gateway_base.py", line 967, in _thread_receiver
    msg = Message.from_io(io)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/execnet/gateway_base.py", line 432, in from_io
    header = io.read(9)  # type 1, channel 4, payload 4
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/execnet/gateway_base.py", line 400, in read
    data = self._read(numbytes - len(buf))

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
--------------------------- Captured stderr teardown ---------------------------
23:02:24.963 | DEBUG   | MainThread   | prefect._internal.concurrency - Draining service <prefect.events.worker.EventsWorker object at 0x7f4a1f735a10>
23:02:24.963 | DEBUG   | MainThread   | prefect._internal.concurrency - Stopping service <prefect.events.worker.EventsWorker object at 0x7f4a1f735a10>
23:02:24.963 | DEBUG   | EventsWorkerThread | prefect._internal.concurrency - Finished call get()
23:02:24.964 | DEBUG   | GlobalEventLoopThread | prefect._internal.concurrency - Exiting service <prefect.events.worker.EventsWorker object at 0x7f4a1f735a10>
23:02:25.156 | ERROR   | prefect.server.services.flowrunnotifications - Error sending notification for policy 0a6a33b6-e862-4d4f-9b52-a2c4a989731e on flow run ea1e7df3-3f1b-4279-a518-f75544131c08
Traceback (most recent call last):
  File "/home/runner/work/prefect/prefect/src/prefect/server/services/flow_run_notifications.py", line 92, in send_flow_run_notification
    block = Block._from_block_document(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/blocks/core.py", line 618, in _from_block_document
    else cls.get_block_class_from_schema(block_document.block_schema)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/blocks/core.py", line 672, in get_block_class_from_schema
    return cls.get_block_class_from_key(block_schema_to_key(schema))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/blocks/core.py", line 683, in get_block_class_from_key
    return lookup_type(cls, key)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/utilities/dispatch.py", line 185, in lookup_type
    raise KeyError(
KeyError: "No class found for dispatch key 'debug-print-notification' in registry for type 'Block'."
23:02:25.344 | ERROR   | prefect.server.services.flowrunnotifications - Unexpected error in: OperationalError('(sqlite3.OperationalError) database is locked')
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1968, in _exec_single_context
    self.dialect.do_execute(
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 113, in execute
    self._adapt_connection._handle_exception(error)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 264, in _handle_exception
    raise error
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 95, in execute
    self.await_(_cursor.execute(operation, parameters))
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 126, in await_only
    return current.driver.switch(awaitable)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 187, in greenlet_spawn
    value = await result
            ^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/cursor.py", line 48, in execute
    await self._execute(self._cursor.execute, sql, parameters)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/cursor.py", line 40, in _execute
    return await self._conn._execute(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/core.py", line 133, in _execute
    return await future
           ^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/core.py", line 106, in run
    result = function()
             ^^^^^^^^^^
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/prefect/prefect/src/prefect/server/services/loop_service.py", line 78, in start
    await self.run_once()
  File "/home/runner/work/prefect/prefect/src/prefect/server/database/dependencies.py", line 119, in async_wrapper
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/server/services/flow_run_notifications.py", line 38, in run_once
    notifications = await db.get_flow_run_notifications_from_queue(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/server/database/interface.py", line 345, in get_flow_run_notifications_from_queue
    return await self.queries.get_flow_run_notifications_from_queue(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/server/database/query_components.py", line 842, in get_flow_run_notifications_from_queue
    await session.execute(delete_stmt)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/ext/asyncio/session.py", line 435, in execute
    result = await greenlet_spawn(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 192, in greenlet_spawn
    result = context.throw(*sys.exc_info())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2232, in execute
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2127, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/orm/bulk_persistence.py", line 1953, in orm_execute_statement
    return super().orm_execute_statement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
    result = conn.execute(
             ^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1413, in execute
    return meth(
           ^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1987, in _exec_single_context
    self._handle_dbapi_exception(
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2344, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1968, in _exec_single_context
    self.dialect.do_execute(
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 113, in execute
    self._adapt_connection._handle_exception(error)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 264, in _handle_exception
    raise error
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 95, in execute
    self.await_(_cursor.execute(operation, parameters))
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 126, in await_only
    return current.driver.switch(awaitable)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 187, in greenlet_spawn
    value = await result
            ^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/cursor.py", line 48, in execute
    await self._execute(self._cursor.execute, sql, parameters)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/cursor.py", line 40, in _execute
    return await self._conn._execute(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/core.py", line 133, in _execute
    return await future
           ^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/core.py", line 106, in run
    result = function()
             ^^^^^^^^^^
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: DELETE FROM flow_run_notification_queue WHERE flow_run_notification_queue.id IN (:id_1_1) RETURNING id]
[parameters: {'id_1_1': '1c763f7f-8f92-4cd0-a0c5-04145fe42786'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
23:02:25.358 | WARNING | prefect.server.services.flowrunnotifications - FlowRunNotifications took 44.367505 seconds to run, which is longer than its loop interval of 4 seconds.
---------------------------- Captured log teardown -----------------------------
ERROR    prefect.server.services.flowrunnotifications:flow_run_notifications.py:113 Error sending notification for policy 0a6a33b6-e862-4d4f-9b52-a2c4a989731e on flow run ea1e7df3-3f1b-4279-a518-f75544131c08
Traceback (most recent call last):
  File "/home/runner/work/prefect/prefect/src/prefect/server/services/flow_run_notifications.py", line 92, in send_flow_run_notification
    block = Block._from_block_document(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/blocks/core.py", line 618, in _from_block_document
    else cls.get_block_class_from_schema(block_document.block_schema)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/blocks/core.py", line 672, in get_block_class_from_schema
    return cls.get_block_class_from_key(block_schema_to_key(schema))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/blocks/core.py", line 683, in get_block_class_from_key
    return lookup_type(cls, key)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/utilities/dispatch.py", line 185, in lookup_type
    raise KeyError(
KeyError: "No class found for dispatch key 'debug-print-notification' in registry for type 'Block'."
ERROR    prefect.server.services.flowrunnotifications:loop_service.py:85 Unexpected error in: OperationalError('(sqlite3.OperationalError) database is locked')
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1968, in _exec_single_context
    self.dialect.do_execute(
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 113, in execute
    self._adapt_connection._handle_exception(error)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 264, in _handle_exception
    raise error
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 95, in execute
    self.await_(_cursor.execute(operation, parameters))
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 126, in await_only
    return current.driver.switch(awaitable)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 187, in greenlet_spawn
    value = await result
            ^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/cursor.py", line 48, in execute
    await self._execute(self._cursor.execute, sql, parameters)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/cursor.py", line 40, in _execute
    return await self._conn._execute(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/core.py", line 133, in _execute
    return await future
           ^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/core.py", line 106, in run
    result = function()
             ^^^^^^^^^^
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/prefect/prefect/src/prefect/server/services/loop_service.py", line 78, in start
    await self.run_once()
  File "/home/runner/work/prefect/prefect/src/prefect/server/database/dependencies.py", line 119, in async_wrapper
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/server/services/flow_run_notifications.py", line 38, in run_once
    notifications = await db.get_flow_run_notifications_from_queue(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/server/database/interface.py", line 345, in get_flow_run_notifications_from_queue
    return await self.queries.get_flow_run_notifications_from_queue(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/prefect/prefect/src/prefect/server/database/query_components.py", line 842, in get_flow_run_notifications_from_queue
    await session.execute(delete_stmt)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/ext/asyncio/session.py", line 435, in execute
    result = await greenlet_spawn(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 192, in greenlet_spawn
    result = context.throw(*sys.exc_info())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2232, in execute
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2127, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/orm/bulk_persistence.py", line 1953, in orm_execute_statement
    return super().orm_execute_statement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
    result = conn.execute(
             ^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1413, in execute
    return meth(
           ^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1987, in _exec_single_context
    self._handle_dbapi_exception(
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2344, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1968, in _exec_single_context
    self.dialect.do_execute(
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 113, in execute
    self._adapt_connection._handle_exception(error)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 264, in _handle_exception
    raise error
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 95, in execute
    self.await_(_cursor.execute(operation, parameters))
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 126, in await_only
    return current.driver.switch(awaitable)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 187, in greenlet_spawn
    value = await result
            ^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/cursor.py", line 48, in execute
    await self._execute(self._cursor.execute, sql, parameters)
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/cursor.py", line 40, in _execute
    return await self._conn._execute(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/core.py", line 133, in _execute
    return await future
           ^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/aiosqlite/core.py", line 106, in run
    result = function()
             ^^^^^^^^^^
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: DELETE FROM flow_run_notification_queue WHERE flow_run_notification_queue.id IN (:id_1_1) RETURNING id]
[parameters: {'id_1_1': '1c763f7f-8f92-4cd0-a0c5-04145fe42786'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
WARNING  prefect.server.services.flowrunnotifications:loop_service.py:92 FlowRunNotifications took 44.367505 seconds to run, which is longer than its loop interval of 4 seconds.

Python 3.11 SQLite

Impact

Example https://github.com/PrefectHQ/prefect/actions/runs/5194057769/jobs/9365329943?pr=9835

Additional context

No response

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 30 days with no activity. To keep this issue open remove stale label or comment.

github-actions[bot] commented 1 year ago

This issue was closed because it has been stale for 14 days with no activity. If this issue is important or you have more to add feel free to re-open it.