Open z63 opened 5 months ago
Thanks for the issue @z63! Can you share info about the size of your DB instance and the volume of data in it?
Hello @desertaxle , on our db instance, there is about 47GB left on filesystem and prefect DB is about 257 MB. The DB is only used by Prefect. Postgres conf was not modified.
Hello, I am still experiencing some DB issues with the same exception in server's log. But this time :
UI : Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Server error '500 Internal Server Error' for url 'https://server.fr/api/flow_runs/55075796-51c1-41b9-8782-5be810370846/set_state' Response: {'exception_message': 'Internal Server Error'} For more information check:https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
Server :
Apr 3 02:21:45 server prefect[1676153]: Traceback (most recent call last):
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 412, in run_asgi
Apr 3 02:21:45 server prefect[1676153]: result = await app( # type: ignore[func-returns-value]
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
Apr 3 02:21:45 server prefect[1676153]: return await self.app(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/applications.py", line 293, in call
Apr 3 02:21:45 server prefect[1676153]: await super().call(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/applications.py", line 123, in call
Apr 3 02:21:45 server prefect[1676153]: await self.middleware_stack(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 190, in call
Apr 3 02:21:45 server prefect[1676153]: raise exc
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 168, in call
Apr 3 02:21:45 server prefect[1676153]: await self.app(scope, receive, _send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/cors.py", line 83, in call
Apr 3 02:21:45 server prefect[1676153]: await self.app(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/exceptions.py", line 62, in call
Apr 3 02:21:45 server prefect[1676153]: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 62, in wrapped_app
Apr 3 02:21:45 server prefect[1676153]: raise exc
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 51, in wrapped_app
Apr 3 02:21:45 server prefect[1676153]: await app(scope, receive, sender)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 20, in call
Apr 3 02:21:45 server prefect[1676153]: raise e
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 17, in call
Apr 3 02:21:45 server prefect[1676153]: await self.app(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 760, in call
Apr 3 02:21:45 server prefect[1676153]: await self.middleware_stack(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 780, in app
Apr 3 02:21:45 server prefect[1676153]: await route.handle(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 483, in handle
Apr 3 02:21:45 server prefect[1676153]: await self.app(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/applications.py", line 293, in call
Apr 3 02:21:45 server prefect[1676153]: await super().call(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/applications.py", line 123, in call
Apr 3 02:21:45 server prefect[1676153]: await self.middleware_stack(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 190, in call
Apr 3 02:21:45 server prefect[1676153]: raise exc
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 168, in call
Apr 3 02:21:45 server prefect[1676153]: await self.app(scope, receive, _send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/gzip.py", line 24, in call
Apr 3 02:21:45 server prefect[1676153]: await responder(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/gzip.py", line 44, in call
Apr 3 02:21:45 server prefect[1676153]: await self.app(scope, receive, self.send_with_gzip)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/exceptions.py", line 62, in call
Apr 3 02:21:45 server prefect[1676153]: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 62, in wrapped_app
Apr 3 02:21:45 server prefect[1676153]: raise exc
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 51, in wrapped_app
Apr 3 02:21:45 server prefect[1676153]: await app(scope, receive, sender)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 20, in call
Apr 3 02:21:45 server prefect[1676153]: raise e
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 17, in call
Apr 3 02:21:45 server prefect[1676153]: await self.app(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 760, in call
Apr 3 02:21:45 server prefect[1676153]: await self.middleware_stack(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 780, in app
Apr 3 02:21:45 server prefect[1676153]: await route.handle(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 302, in handle
Apr 3 02:21:45 server prefect[1676153]: await self.app(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 81, in app
Apr 3 02:21:45 server prefect[1676153]: await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 62, in wrapped_app
Apr 3 02:21:45 server prefect[1676153]: raise exc
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 51, in wrapped_app
Apr 3 02:21:45 server prefect[1676153]: await app(scope, receive, sender)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 79, in app
Apr 3 02:21:45 server prefect[1676153]: await response(scope, receive, send)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/responses.py", line 155, in call
Apr 3 02:21:45 server prefect[1676153]: await self.background()
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/background.py", line 43, in call
Apr 3 02:21:45 server prefect[1676153]: await task()
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/background.py", line 26, in call
Apr 3 02:21:45 server prefect[1676153]: await self.func(*self.args, **self.kwargs)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/server/api/workers.py", line 375, in _record_work_queue_polls
Apr 3 02:21:45 server prefect[1676153]: await models.workers.update_work_queue(
Apr 3 02:21:45 server prefect[1676153]: File "/usr/lib/python3.9/contextlib.py", line 182, in aexit
Apr 3 02:21:45 server prefect[1676153]: await self.gen.anext()
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/server/database/interface.py", line 119, in session_context
Apr 3 02:21:45 server prefect[1676153]: yield session
Apr 3 02:21:45 server prefect[1676153]: File "/usr/lib/python3.9/contextlib.py", line 182, in aexit
Apr 3 02:21:45 server prefect[1676153]: await self.gen.anext()
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/server/database/configurations.py", line 277, in begin_transaction
Apr 3 02:21:45 server prefect[1676153]: yield transaction
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/ext/asyncio/session.py", line 1867, in aexit
Apr 3 02:21:45 server prefect[1676153]: await greenlet_spawn(
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 199, in greenlet_spawn
Apr 3 02:21:45 server prefect[1676153]: result = context.switch(value)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/util.py", line 148, in exit
Apr 3 02:21:45 server prefect[1676153]: self.rollback()
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 146, in exit
Apr 3 02:21:45 server prefect[1676153]: raise exc_value.with_traceback(exc_tb)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/util.py", line 144, in exit
Apr 3 02:21:45 server prefect[1676153]: self.commit()
Apr 3 02:21:45 server prefect[1676153]: File "<string>", line 2, in commit
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
Apr 3 02:21:45 server prefect[1676153]: ret_value = fn(self, *arg, **kw)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1264, in commit
Apr 3 02:21:45 server prefect[1676153]: trans.commit()
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2611, in commit
Apr 3 02:21:45 server prefect[1676153]: self._do_commit()
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2716, in _do_commit
Apr 3 02:21:45 server prefect[1676153]: self._connection_commit_impl()
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2687, in _connection_commit_impl
Apr 3 02:21:45 server prefect[1676153]: self.connection._commit_impl()
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1136, in _commit_impl
Apr 3 02:21:45 server prefect[1676153]: self._handle_dbapi_exception(e, None, None, None, None)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2338, in _handle_dbapi_exception
Apr 3 02:21:45 server prefect[1676153]: raise exc_info[1].with_traceback(exc_info[2])
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1134, in _commit_impl
Apr 3 02:21:45 server prefect[1676153]: self.engine.dialect.do_commit(self.connection)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 695, in do_commit
Apr 3 02:21:45 server prefect[1676153]: dbapi_connection.commit()
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 867, in commit
Apr 3 02:21:45 server prefect[1676153]: self._handle_exception(error)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 791, in handle_exception
Apr 3 02:21:45 server prefect[1676153]: raise error
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 865, in commit
Apr 3 02:21:45 server prefect[1676153]: self.await(self._transaction.commit())
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 127, in await_only
Apr 3 02:21:45 server prefect[1676153]: return current.driver.switch(awaitable) # type: ignore[no-any-return]
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 192, in greenlet_spawn
Apr 3 02:21:45 server prefect[1676153]: value = await result
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/asyncpg/transaction.py", line 219, in commit
Apr 3 02:21:45 server prefect[1676153]: await self.__commit()
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/asyncpg/transaction.py", line 187, in __commit
Apr 3 02:21:45 server prefect[1676153]: await self._connection.execute(query)
Apr 3 02:21:45 server prefect[1676153]: File "/etc/prefect/venv/lib/python3.9/site-packages/asyncpg/connection.py", line 350, in execute
Apr 3 02:21:45 server prefect[1676153]: result = await self._protocol.query(query, timeout)
Apr 3 02:21:45 server prefect[1676153]: File "asyncpg/protocol/protocol.pyx", line 374, in query
Apr 3 02:21:45 server prefect[1676153]: asyncio.exceptions.TimeoutError
2024-04-03 02:21:21.399 CEST [1682943] prefect@prefect ERROR: canceling statement due to user request
2024-04-03 02:21:21.399 CEST [1682943] prefect@prefect CONTEXT: while updating tuple (0,37) in relation "work_queue"
2024-04-03 02:21:21.399 CEST [1682943] prefect@prefect STATEMENT: UPDATE work_queue SET last_polled=$1::TIMESTAMP WITH TIME ZONE, updated=CURRENT_TIMESTAMP WHERE work_queue.id = $2::UUID
2024-04-03 02:21:27.728 CEST [1687175] prefect@prefect ERROR: canceling statement due to user request
2024-04-03 02:21:27.728 CEST [1687175] prefect@prefect CONTEXT: while updating tuple (0,38) in relation "work_queue"
2024-04-03 02:21:27.728 CEST [1687175] prefect@prefect STATEMENT: UPDATE work_queue SET last_polled=$1::TIMESTAMP WITH TIME ZONE, updated=CURRENT_TIMESTAMP WHERE work_queue.id = $2::UUID
2024-04-03 02:21:40.749 CEST [1687202] prefect@prefect ERROR: canceling statement due to user request
2024-04-03 02:21:40.749 CEST [1687202] prefect@prefect CONTEXT: while rechecking updated tuple (0,39) in relation "work_queue"
2024-04-03 02:21:40.749 CEST [1687202] prefect@prefect STATEMENT: UPDATE work_queue SET last_polled=$1::TIMESTAMP WITH TIME ZONE, updated=CURRENT_TIMESTAMP WHERE work_queue.id = $2::UUID
First check
Bug summary
I get random PrefectHTTPStatusError visible UI causing the flow to fail. I check the server's log. It shows a asyncio.exceptions.TimeoutError repeated several times.
I first workaround the issue by increasing PREFECT_API_DATABASE_CONNECTION_TIMEOUT to 10s and PREFECT_API_DATABASE_TIMEOUT to 20s. I noticed improvement because I have this error several times a day before workaround to 0 error during about 7 days.
But then, I have got this error back. Except upgrading to latest prefect release, I do nothing special in this interval in my deployments/flows, I do not identify some special process in our flow causing this error. I use a postgres server dedicated to prefect. I have 2 workers, one is on the same machine that prefect server and postgres. Errors may happen with 2 workers.
Reproduction
Error
Versions
Additional context
No response