Any calls to logging.dictConfig including from 3rd-party libraries (in my case ray) raises an exception inside prefect APILogger.
File "/deps/prefect/logging/handlers.py", line 104, in flush
RuntimeError: Cannot call `APILogWorker.flush` from the global event loop; it would block the event loop and cause a deadlock. Use `APILogWorker.aflush` instead.
create a deployment from this and run it such that it goes through python -m prefect.engine. Once the module tries to load it will configures logging via logging.dictConfig which internally will call .flush() on the handlers of the root logger. Since APILogger is one of them, it will fail since we are calling from the global thread since we have not yet created the thread where the flow is running in.
Running the flow directly via the script has no problems but this is not how deployments work.
Version info (prefect version output)
any prefect version <3
python 3.10
Additional context
Original error message
Traceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/src/main.py", line 30, in <module>
import ray # noqa: F401
File "/deps/ray/__init__.py", line 7, in <module>
log.generate_logging_config()
File "/deps/ray/_private/log.py", line 109, in generate_logging_config
dictConfig(
File "/usr/local/lib/python3.10/logging/config.py", line 811, in dictConfig
dictConfigClass(config).configure()
File "/usr/local/lib/python3.10/logging/config.py", line 538, in configure
_clearExistingHandlers()
File "/usr/local/lib/python3.10/logging/config.py", line 275, in _clearExistingHandlers
logging.shutdown(logging._handlerList[:])
File "/usr/local/lib/python3.10/logging/__init__.py", line 2182, in shutdown
h.flush()
File "/deps/prefect/logging/handlers.py", line 104, in flush
raise RuntimeError(
RuntimeError: Cannot call `APILogWorker.flush` from the global event loop; it would block the event loop and cause a deadlock. Use `APILogWorker.aflush` instead.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/deps/prefect/engine.py", line 420, in retrieve_flow_then_begin_flow_run
load_flow_from_entrypoint(entrypoint)
File "/deps/prefect/flows.py", line 1685, in load_flow_from_entrypoint
flow = import_object(entrypoint)
File "/deps/prefect/utilities/importtools.py", line 201, in import_object
module = load_script_as_module(script_path)
File "/deps/prefect/utilities/importtools.py", line 164, in load_script_as_module
raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at '/src/tools/dataset_generator/main.py' encountered an exception: RuntimeError('Cannot call `APILogWorker.flush` from the global event loop; it would block the event loop and cause a deadlock. Use `APILogWorker.aflush` instead.')
Bug summary
Any calls to logging.dictConfig including from 3rd-party libraries (in my case ray) raises an exception inside prefect APILogger.
Reproduce:
create a deployment from this and run it such that it goes through
python -m prefect.engine
. Once the module tries to load it will configures logging via logging.dictConfig which internally will call .flush() on the handlers of the root logger. Since APILogger is one of them, it will fail since we are calling from the global thread since we have not yet created the thread where the flow is running in.Running the flow directly via the script has no problems but this is not how deployments work.
Version info (
prefect version
output)Additional context
Original error message