Deleting multiple flow runs from the UI fails with “Failed to delete Flow Run”

l3hgs-brent commented 4 days ago

Bug summary

A “Failed to delete Flow Run” error is received on the UI Runs page and clicking the delete icon button (see screenshot below).

Typically, only between one and a few of the selected runs are deleted when this occurs.

Version info

Version:             3.1.3
API version:         0.8.4
Python version:      3.9.13
Git commit:          39b6028c
Built:               Tue, Nov 19, 2024 3:25 PM
OS/Arch:             win32/AMD64
Profile:             default
Server type:         server
Pydantic version:    2.7.4

Additional context

zzstoatzz commented 4 days ago

hi @l3hgs-brent - thank you for the issue! do you mind sharing if there was anything in your browser console / network tab?

l3hgs-brent commented 4 days ago

hi @l3hgs-brent - thank you for the issue! do you mind sharing if there was anything in your browser console / network tab?

With four flow runs selected on the Runs page, when clicking delete, two of the four were deleted and the browser console shows two 503 errors ...

   DELETE http://127.0.0.1:4200/api/flow_runs/0d48445f-0de5-49fa-993d-3712024119b5 503 (Service Unavailable)       
   DELETE http://127.0.0.1:4200/api/flow_runs/a95080ef-5b46-4ac6-b6c0-5cbd97568c22 503 (Service Unavailable)

Both of these runs could be subsequently individually deleted.

zzstoatzz commented 3 days ago

ok @l3hgs-brent - thanks for the context. I think this was likely an intermittent service disruption with Prefect Cloud of some kind, and likely not so much a UI issue. I'll defer to @pleek91 on whether its worth it / prudent to retry or be more informative in situations like this

l3hgs-brent commented 2 days ago

We're running a local Prefect Core server. It seems the following SQLite "database is locked" error is logged when this occurs ...

06:40:07.362 | ERROR   | uvicorn.error - Exception in ASGI application
Traceback (most recent call last):
  File "C:\Users\prefect\AppData\Local\Programs\Python\Python39\lib\site-packages\sqlalchemy\engine\base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
...
  File "C:\Users\prefect\AppData\Local\Programs\Python\Python39\lib\site-packages\aiosqlite\core.py", line 115, in run
    result = function()
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\prefect\AppData\Local\Programs\Python\Python39\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
...
  File "C:\Users\prefect\AppData\Local\Programs\Python\Python39\lib\site-packages\aiosqlite\core.py", line 115, in run
    result = function()
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: UPDATE deployment SET last_polled=:last_polled, status=:status, updated=strftime('%Y-%m-%d %H:%M:%f000', 'now') WHERE deployment.id IN (SELECT 1 FROM (SELECT 1) WHERE 1!=1) OR deployment.work_queue_id IN (:work_queue_id_1_1)]
[parameters: {'last_polled': '2024-11-27 13:40:07.360157', 'status': 'READY', 'work_queue_id_1_1': 'd9716c16-a2a8-4c57-a8fc-d9ca2cd75f75'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
06:40:07.389 | ERROR   | uvicorn.error - Exception in ASGI application
Traceback (most recent call last):
  File "C:\Users\prefect\AppData\Local\Programs\Python\Python39\lib\site-packages\sqlalchemy\engine\base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
..
  File "C:\Users\prefect\AppData\Local\Programs\Python\Python39\lib\site-packages\aiosqlite\core.py", line 115, in run
    result = function()
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\prefect\AppData\Local\Programs\Python\Python39\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
...
  File "C:\Users\prefect\AppData\Local\Programs\Python\Python39\lib\site-packages\aiosqlite\core.py", line 115, in run
    result = function()
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: DELETE FROM flow_run WHERE flow_run.id = :id_1]
[parameters: {'id_1': 'abfeae93-429a-49a9-a382-5b77ff0dd649'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

There do appear to be a number of GitHub issues mentioning 'database is locked', with a couple still open. For whatever it's worth, we just recently upgraded to Prefect 3 and don't recall seeing this problem with Prefect 2.

PrefectHQ / prefect