search

PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.77k stars 1.54k forks source link

Random 500 Internal Server Error in UI #12442

Open z63 opened 5 months ago

z63 commented 5 months ago

First check

Bug summary

I get random PrefectHTTPStatusError visible UI causing the flow to fail. I check the server's log. It shows a asyncio.exceptions.TimeoutError repeated several times.

I first workaround the issue by increasing PREFECT_API_DATABASE_CONNECTION_TIMEOUT to 10s and PREFECT_API_DATABASE_TIMEOUT to 20s. I noticed improvement because I have this error several times a day before workaround to 0 error during about 7 days.

But then, I have got this error back. Except upgrading to latest prefect release, I do nothing special in this interval in my deployments/flows, I do not identify some special process in our flow causing this error. I use a postgres server dedicated to prefect. I have 2 workers, one is on the same machine that prefect server and postgres. Errors may happen with 2 workers.

Reproduction

It is a random bug

Error

Server side : 

Mar 27 02:19:36 server prefect[1477307]: 02:19:36.179 | ERROR   | prefect.server - Encountered exception in request:
Mar 27 02:19:36 server prefect[1477307]: Traceback (most recent call last):
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 168, in __call__
Mar 27 02:19:36 server prefect[1477307]:     await self.app(scope, receive, _send)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/gzip.py", line 24, in __call__
Mar 27 02:19:36 server prefect[1477307]:     await responder(scope, receive, send)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/gzip.py", line 44, in __call__
Mar 27 02:19:36 server prefect[1477307]:     await self.app(scope, receive, self.send_with_gzip)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/exceptions.py", line 62, in __call__
Mar 27 02:19:36 server prefect[1477307]:     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 62, in wrapped_app
Mar 27 02:19:36 server prefect[1477307]:     raise exc
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 51, in wrapped_app
Mar 27 02:19:36 server prefect[1477307]:     await app(scope, receive, sender)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 20, in __call__
Mar 27 02:19:36 server prefect[1477307]:     raise e
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 17, in __call__
Mar 27 02:19:36 server prefect[1477307]:     await self.app(scope, receive, send)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 760, in __call__
Mar 27 02:19:36 server prefect[1477307]:     await self.middleware_stack(scope, receive, send)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 780, in app
Mar 27 02:19:36 server prefect[1477307]:     await route.handle(scope, receive, send)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 302, in handle
Mar 27 02:19:36 server prefect[1477307]:     await self.app(scope, receive, send)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 81, in app
Mar 27 02:19:36 server prefect[1477307]:     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 62, in wrapped_app
Mar 27 02:19:36 server prefect[1477307]:     raise exc
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 51, in wrapped_app
Mar 27 02:19:36 server prefect[1477307]:     await app(scope, receive, sender)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 76, in app
Mar 27 02:19:36 server prefect[1477307]:     response = await func(request)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/server/utilities/server.py", line 104, in handle_response_scoped_depends
Mar 27 02:19:36 server prefect[1477307]:     response = await default_handler(request)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/routing.py", line 251, in app
Mar 27 02:19:36 server prefect[1477307]:     raw_response = await run_endpoint_function(
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/routing.py", line 177, in run_endpoint_function
Mar 27 02:19:36 server prefect[1477307]:     return await dependant.call(**values)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/server/api/workers.py", line 569, in worker_heartbeat
Mar 27 02:19:36 server prefect[1477307]:     await models.workers.worker_heartbeat(
Mar 27 02:19:36 server prefect[1477307]:   File "/usr/lib/python3.9/contextlib.py", line 182, in __aexit__
Mar 27 02:19:36 server prefect[1477307]:     await self.gen.__anext__()
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/server/database/interface.py", line 119, in session_context
Mar 27 02:19:36 server prefect[1477307]:     yield session
Mar 27 02:19:36 server prefect[1477307]:   File "/usr/lib/python3.9/contextlib.py", line 182, in __aexit__
Mar 27 02:19:36 server prefect[1477307]:     await self.gen.__anext__()
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/server/database/configurations.py", line 277, in begin_transaction
Mar 27 02:19:36 server prefect[1477307]:     yield transaction
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/ext/asyncio/session.py", line 1867, in __aexit__
Mar 27 02:19:36 server prefect[1477307]:     await greenlet_spawn(
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 199, in greenlet_spawn
Mar 27 02:19:36 server prefect[1477307]:     result = context.switch(value)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/util.py", line 148, in __exit__
Mar 27 02:19:36 server prefect[1477307]:     self.rollback()
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
Mar 27 02:19:36 server prefect[1477307]:     raise exc_value.with_traceback(exc_tb)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/util.py", line 144, in __exit__
Mar 27 02:19:36 server prefect[1477307]:     self.commit()
Mar 27 02:19:36 server prefect[1477307]:   File "<string>", line 2, in commit
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
Mar 27 02:19:36 server prefect[1477307]:     ret_value = fn(self, *arg, **kw)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1264, in commit
Mar 27 02:19:36 server prefect[1477307]:     trans.commit()
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2611, in commit
Mar 27 02:19:36 server prefect[1477307]:     self._do_commit()
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2716, in _do_commit
Mar 27 02:19:36 server prefect[1477307]:     self._connection_commit_impl()
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2687, in _connection_commit_impl
Mar 27 02:19:36 server prefect[1477307]:     self.connection._commit_impl()
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1136, in _commit_impl
Mar 27 02:19:36 server prefect[1477307]:     self._handle_dbapi_exception(e, None, None, None, None)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2338, in _handle_dbapi_exception
Mar 27 02:19:36 server prefect[1477307]:     raise exc_info[1].with_traceback(exc_info[2])
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1134, in _commit_impl
Mar 27 02:19:36 server prefect[1477307]:     self.engine.dialect.do_commit(self.connection)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 695, in do_commit
Mar 27 02:19:36 server prefect[1477307]:     dbapi_connection.commit()
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 867, in commit
Mar 27 02:19:36 server prefect[1477307]:     self._handle_exception(error)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 791, in _handle_exception
Mar 27 02:19:36 server prefect[1477307]:     raise error
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 865, in commit
Mar 27 02:19:36 server prefect[1477307]:     self.await_(self._transaction.commit())
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 127, in await_only
Mar 27 02:19:36 server prefect[1477307]:     return current.driver.switch(awaitable)  # type: ignore[no-any-return]
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 192, in greenlet_spawn
Mar 27 02:19:36 server prefect[1477307]:     value = await result
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/asyncpg/transaction.py", line 219, in commit
Mar 27 02:19:36 server prefect[1477307]:     await self.__commit()
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/asyncpg/transaction.py", line 187, in __commit
Mar 27 02:19:36 server prefect[1477307]:     await self._connection.execute(query)
Mar 27 02:19:36 server prefect[1477307]:   File "/etc/prefect/venv/lib/python3.9/site-packages/asyncpg/connection.py", line 350, in execute
Mar 27 02:19:36 server prefect[1477307]:     result = await self._protocol.query(query, timeout)
Mar 27 02:19:36 server prefect[1477307]:   File "asyncpg/protocol/protocol.pyx", line 374, in query
Mar 27 02:19:36 server prefect[1477307]: asyncio.exceptions.TimeoutError

Worker side : 

Encountered exception during execution:
Traceback (most recent call last):
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/engine.py", line 867, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
    return await asyncio.wrap_future(self.future)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "all.py", line 109, in elt
    transform_vault(
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/flows.py", line 1224, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/engine.py", line 297, in enter_flow_run_engine_from_flow_call
    retval = from_sync.wait_for_call_in_loop_thread(
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/api.py", line 243, in wait_for_call_in_loop_thread
    return call.result()
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/client/utilities.py", line 78, in with_injected_client
    return await fn(*args, **kwargs)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/engine.py", line 749, in create_and_begin_subflow_run
    return await terminal_state.result(fetch=True)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/states.py", line 91, in _get_state_result
    raise await get_state_exception(state)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/engine.py", line 867, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
    return await asyncio.wrap_future(self.future)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/tmp/tmpielqmq2oprefect/my-project-main/transform/transform.py", line 99, in transform_vault
    create_stuff()
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/flows.py", line 1224, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/engine.py", line 297, in enter_flow_run_engine_from_flow_call
    retval = from_sync.wait_for_call_in_loop_thread(
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/api.py", line 243, in wait_for_call_in_loop_thread
    return call.result()
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/client/utilities.py", line 78, in with_injected_client
    return await fn(*args, **kwargs)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/engine.py", line 749, in create_and_begin_subflow_run
    return await terminal_state.result(fetch=True)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/states.py", line 91, in _get_state_result
    raise await get_state_exception(state)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/engine.py", line 867, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
    return await asyncio.wrap_future(self.future)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/tmp/tmpielqmq2oprefect/my-project-main/transform/business_logic.py", line 13, in create_stuff
    create_and_populate_stuff()
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/flows.py", line 1224, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/engine.py", line 297, in enter_flow_run_engine_from_flow_call
    retval = from_sync.wait_for_call_in_loop_thread(
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/api.py", line 243, in wait_for_call_in_loop_thread
    return call.result()
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/client/utilities.py", line 78, in with_injected_client
    return await fn(*args, **kwargs)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/engine.py", line 749, in create_and_begin_subflow_run
    return await terminal_state.result(fetch=True)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/states.py", line 91, in _get_state_result
    raise await get_state_exception(state)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/engine.py", line 867, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
    return await asyncio.wrap_future(self.future)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/tmp/tmpielqmq2oprefect/my-project-main/transform/business_logic.py", line 265, in create_and_populate_stuff
    create_and_insert_stuff()
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/flows.py", line 1224, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/engine.py", line 297, in enter_flow_run_engine_from_flow_call
    retval = from_sync.wait_for_call_in_loop_thread(
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/api.py", line 243, in wait_for_call_in_loop_thread
    return call.result()
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/client/utilities.py", line 78, in with_injected_client
    return await fn(*args, **kwargs)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/engine.py", line 657, in create_and_begin_subflow_run
    flow_run = await client.create_flow_run(
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/client/orchestration.py", line 662, in create_flow_run
    response = await self._client.post("/flow_runs/", json=flow_run_create_json)
  File "/etc/prefect/venv/lib/python3.9/site-packages/httpx/_client.py", line 1892, in post
    return await self.request(
  File "/etc/prefect/venv/lib/python3.9/site-packages/httpx/_client.py", line 1574, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/client/base.py", line 344, in send
    response.raise_for_status()
  File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/client/base.py", line 167, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url 'https://server.prd.company.fr/api/flow_runs/'
Response: {'exception_message': 'Internal Server Error'}
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500

Versions

Version:             2.16.6
API version:         0.8.4
Python version:      3.10.12
Git commit:          3fecd435
Built:               Sat, Mar 23, 2024 4:06 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.37.2

Additional context

No response

desertaxle commented 5 months ago

Thanks for the issue @z63! Can you share info about the size of your DB instance and the volume of data in it?

z63 commented 5 months ago

Hello @desertaxle , on our db instance, there is about 47GB left on filesystem and prefect DB is about 257 MB. The DB is only used by Prefect. Postgres conf was not modified.

z63 commented 5 months ago

Hello, I am still experiencing some DB issues with the same exception in server's log. But this time :

UI : Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Server error '500 Internal Server Error' for url 'https://server.fr/api/flow_runs/55075796-51c1-41b9-8782-5be810370846/set_state' Response: {'exception_message': 'Internal Server Error'} For more information check:https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500

Server : 

Apr  3 02:21:45 server prefect[1676153]: Traceback (most recent call last):
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 412, in run_asgi
Apr  3 02:21:45 server prefect[1676153]:     result = await app(  # type: ignore[func-returns-value]
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
Apr  3 02:21:45 server prefect[1676153]:     return await self.app(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/applications.py", line 293, in call
Apr  3 02:21:45 server prefect[1676153]:     await super().call(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/applications.py", line 123, in call
Apr  3 02:21:45 server prefect[1676153]:     await self.middleware_stack(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 190, in call
Apr  3 02:21:45 server prefect[1676153]:     raise exc
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 168, in call
Apr  3 02:21:45 server prefect[1676153]:     await self.app(scope, receive, _send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/cors.py", line 83, in call
Apr  3 02:21:45 server prefect[1676153]:     await self.app(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/exceptions.py", line 62, in call
Apr  3 02:21:45 server prefect[1676153]:     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 62, in wrapped_app
Apr  3 02:21:45 server prefect[1676153]:     raise exc
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 51, in wrapped_app
Apr  3 02:21:45 server prefect[1676153]:     await app(scope, receive, sender)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 20, in call
Apr  3 02:21:45 server prefect[1676153]:     raise e
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 17, in call
Apr  3 02:21:45 server prefect[1676153]:     await self.app(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 760, in call
Apr  3 02:21:45 server prefect[1676153]:     await self.middleware_stack(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 780, in app
Apr  3 02:21:45 server prefect[1676153]:     await route.handle(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 483, in handle
Apr  3 02:21:45 server prefect[1676153]:     await self.app(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/applications.py", line 293, in call
Apr  3 02:21:45 server prefect[1676153]:     await super().call(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/applications.py", line 123, in call
Apr  3 02:21:45 server prefect[1676153]:     await self.middleware_stack(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 190, in call
Apr  3 02:21:45 server prefect[1676153]:     raise exc
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 168, in call
Apr  3 02:21:45 server prefect[1676153]:     await self.app(scope, receive, _send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/gzip.py", line 24, in call
Apr  3 02:21:45 server prefect[1676153]:     await responder(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/gzip.py", line 44, in call
Apr  3 02:21:45 server prefect[1676153]:     await self.app(scope, receive, self.send_with_gzip)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/middleware/exceptions.py", line 62, in call
Apr  3 02:21:45 server prefect[1676153]:     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 62, in wrapped_app
Apr  3 02:21:45 server prefect[1676153]:     raise exc
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 51, in wrapped_app
Apr  3 02:21:45 server prefect[1676153]:     await app(scope, receive, sender)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 20, in call
Apr  3 02:21:45 server prefect[1676153]:     raise e
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 17, in call
Apr  3 02:21:45 server prefect[1676153]:     await self.app(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 760, in call
Apr  3 02:21:45 server prefect[1676153]:     await self.middleware_stack(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 780, in app
Apr  3 02:21:45 server prefect[1676153]:     await route.handle(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 302, in handle
Apr  3 02:21:45 server prefect[1676153]:     await self.app(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 81, in app
Apr  3 02:21:45 server prefect[1676153]:     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 62, in wrapped_app
Apr  3 02:21:45 server prefect[1676153]:     raise exc
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 51, in wrapped_app
Apr  3 02:21:45 server prefect[1676153]:     await app(scope, receive, sender)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/routing.py", line 79, in app
Apr  3 02:21:45 server prefect[1676153]:     await response(scope, receive, send)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/responses.py", line 155, in call
Apr  3 02:21:45 server prefect[1676153]:     await self.background()
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/background.py", line 43, in call
Apr  3 02:21:45 server prefect[1676153]:     await task()
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/_vendor/starlette/background.py", line 26, in call
Apr  3 02:21:45 server prefect[1676153]:     await self.func(*self.args, **self.kwargs)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/server/api/workers.py", line 375, in _record_work_queue_polls
Apr  3 02:21:45 server prefect[1676153]:     await models.workers.update_work_queue(
Apr  3 02:21:45 server prefect[1676153]:   File "/usr/lib/python3.9/contextlib.py", line 182, in aexit
Apr  3 02:21:45 server prefect[1676153]:     await self.gen.anext()
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/server/database/interface.py", line 119, in session_context
Apr  3 02:21:45 server prefect[1676153]:     yield session
Apr  3 02:21:45 server prefect[1676153]:   File "/usr/lib/python3.9/contextlib.py", line 182, in aexit
Apr  3 02:21:45 server prefect[1676153]:     await self.gen.anext()
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/prefect/server/database/configurations.py", line 277, in begin_transaction
Apr  3 02:21:45 server prefect[1676153]:     yield transaction
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/ext/asyncio/session.py", line 1867, in aexit
Apr  3 02:21:45 server prefect[1676153]:     await greenlet_spawn(
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 199, in greenlet_spawn
Apr  3 02:21:45 server prefect[1676153]:     result = context.switch(value)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/util.py", line 148, in exit
Apr  3 02:21:45 server prefect[1676153]:     self.rollback()
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 146, in exit
Apr  3 02:21:45 server prefect[1676153]:     raise exc_value.with_traceback(exc_tb)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/util.py", line 144, in exit
Apr  3 02:21:45 server prefect[1676153]:     self.commit()
Apr  3 02:21:45 server prefect[1676153]:   File "<string>", line 2, in commit
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
Apr  3 02:21:45 server prefect[1676153]:     ret_value = fn(self, *arg, **kw)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1264, in commit
Apr  3 02:21:45 server prefect[1676153]:     trans.commit()
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2611, in commit
Apr  3 02:21:45 server prefect[1676153]:     self._do_commit()
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2716, in _do_commit
Apr  3 02:21:45 server prefect[1676153]:     self._connection_commit_impl()
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2687, in _connection_commit_impl
Apr  3 02:21:45 server prefect[1676153]:     self.connection._commit_impl()
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1136, in _commit_impl
Apr  3 02:21:45 server prefect[1676153]:     self._handle_dbapi_exception(e, None, None, None, None)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2338, in _handle_dbapi_exception
Apr  3 02:21:45 server prefect[1676153]:     raise exc_info[1].with_traceback(exc_info[2])
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1134, in _commit_impl
Apr  3 02:21:45 server prefect[1676153]:     self.engine.dialect.do_commit(self.connection)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 695, in do_commit
Apr  3 02:21:45 server prefect[1676153]:     dbapi_connection.commit()
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 867, in commit
Apr  3 02:21:45 server prefect[1676153]:     self._handle_exception(error)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 791, in handle_exception
Apr  3 02:21:45 server prefect[1676153]:     raise error
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 865, in commit
Apr  3 02:21:45 server prefect[1676153]:     self.await(self._transaction.commit())
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 127, in await_only
Apr  3 02:21:45 server prefect[1676153]:     return current.driver.switch(awaitable)  # type: ignore[no-any-return]
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 192, in greenlet_spawn
Apr  3 02:21:45 server prefect[1676153]:     value = await result
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/asyncpg/transaction.py", line 219, in commit
Apr  3 02:21:45 server prefect[1676153]:     await self.__commit()
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/asyncpg/transaction.py", line 187, in __commit
Apr  3 02:21:45 server prefect[1676153]:     await self._connection.execute(query)
Apr  3 02:21:45 server prefect[1676153]:   File "/etc/prefect/venv/lib/python3.9/site-packages/asyncpg/connection.py", line 350, in execute
Apr  3 02:21:45 server prefect[1676153]:     result = await self._protocol.query(query, timeout)
Apr  3 02:21:45 server prefect[1676153]:   File "asyncpg/protocol/protocol.pyx", line 374, in query
Apr  3 02:21:45 server prefect[1676153]: asyncio.exceptions.TimeoutError
2024-04-03 02:21:21.399 CEST [1682943] prefect@prefect ERROR:  canceling statement due to user request
2024-04-03 02:21:21.399 CEST [1682943] prefect@prefect CONTEXT:  while updating tuple (0,37) in relation "work_queue"
2024-04-03 02:21:21.399 CEST [1682943] prefect@prefect STATEMENT:  UPDATE work_queue SET last_polled=$1::TIMESTAMP WITH TIME ZONE, updated=CURRENT_TIMESTAMP WHERE work_queue.id = $2::UUID
2024-04-03 02:21:27.728 CEST [1687175] prefect@prefect ERROR:  canceling statement due to user request
2024-04-03 02:21:27.728 CEST [1687175] prefect@prefect CONTEXT:  while updating tuple (0,38) in relation "work_queue"
2024-04-03 02:21:27.728 CEST [1687175] prefect@prefect STATEMENT:  UPDATE work_queue SET last_polled=$1::TIMESTAMP WITH TIME ZONE, updated=CURRENT_TIMESTAMP WHERE work_queue.id = $2::UUID
2024-04-03 02:21:40.749 CEST [1687202] prefect@prefect ERROR:  canceling statement due to user request
2024-04-03 02:21:40.749 CEST [1687202] prefect@prefect CONTEXT:  while rechecking updated tuple (0,39) in relation "work_queue"
2024-04-03 02:21:40.749 CEST [1687202] prefect@prefect STATEMENT:  UPDATE work_queue SET last_polled=$1::TIMESTAMP WITH TIME ZONE, updated=CURRENT_TIMESTAMP WHERE work_queue.id = $2::UUID