PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
17.6k stars 1.65k forks source link

Any calls to logging.dictConfig within a Flow raise an error in prefect deployments #15063

Open marcm-ml opened 3 months ago

marcm-ml commented 3 months ago

Bug summary

Any calls to logging.dictConfig including from 3rd-party libraries (in my case ray) raises an exception inside prefect APILogger.

File "/deps/prefect/logging/handlers.py", line 104, in flush
RuntimeError: Cannot call `APILogWorker.flush` from the global event loop; it would block the event loop and cause a deadlock. Use `APILogWorker.aflush` instead.

Reproduce:

import logging.config

from prefect import flow

logging.config.dictConfig({"version": 1})

@flow()
def my_flow():
   print(hi)

create a deployment from this and run it such that it goes through python -m prefect.engine. Once the module tries to load it will configures logging via logging.dictConfig which internally will call .flush() on the handlers of the root logger. Since APILogger is one of them, it will fail since we are calling from the global thread since we have not yet created the thread where the flow is running in.

Running the flow directly via the script has no problems but this is not how deployments work.

Version info (prefect version output)

any prefect version <3
python 3.10

Additional context

Original error message

Traceback (most recent call last):                                                                                                                                  
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module                                                                                           
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed                                                                                      
  File "/src/main.py", line 30, in <module>                                                                                                 
    import ray  # noqa: F401                                                                                                                                        
  File "/deps/ray/__init__.py", line 7, in <module>                                                                                                                 
    log.generate_logging_config()                                                                                                                                   
  File "/deps/ray/_private/log.py", line 109, in generate_logging_config                                                                                            
    dictConfig(                                                                                                                                                     
  File "/usr/local/lib/python3.10/logging/config.py", line 811, in dictConfig                                                                                       
    dictConfigClass(config).configure()                                                                                                                             
  File "/usr/local/lib/python3.10/logging/config.py", line 538, in configure                                                                                        
    _clearExistingHandlers()                                                                                                                                        
  File "/usr/local/lib/python3.10/logging/config.py", line 275, in _clearExistingHandlers                                                                           
    logging.shutdown(logging._handlerList[:])                                                                                                                       
  File "/usr/local/lib/python3.10/logging/__init__.py", line 2182, in shutdown                                                                                      
    h.flush()                                                                                                                                                       
  File "/deps/prefect/logging/handlers.py", line 104, in flush                                                                                                      
    raise RuntimeError(                                                                                                                                             
RuntimeError: Cannot call `APILogWorker.flush` from the global event loop; it would block the event loop and cause a deadlock. Use `APILogWorker.aflush` instead.   

The above exception was the direct cause of the following exception:                                                                                                

Traceback (most recent call last):                                                                                                                                  
  File "/deps/prefect/engine.py", line 420, in retrieve_flow_then_begin_flow_run                                                                                    
    load_flow_from_entrypoint(entrypoint)                                                                                                                           
  File "/deps/prefect/flows.py", line 1685, in load_flow_from_entrypoint                                                                                            
    flow = import_object(entrypoint)                                                                                                                                
  File "/deps/prefect/utilities/importtools.py", line 201, in import_object                                                                                         
    module = load_script_as_module(script_path)                                                                                                                     
  File "/deps/prefect/utilities/importtools.py", line 164, in load_script_as_module                                                                                 
    raise ScriptError(user_exc=exc, path=path) from exc                                                                                                             
prefect.exceptions.ScriptError: Script at '/src/tools/dataset_generator/main.py' encountered an exception: RuntimeError('Cannot call `APILogWorker.flush` from the  global event loop; it would block the event loop and cause a deadlock. Use `APILogWorker.aflush` instead.')
marcm-ml commented 2 months ago

Update: this is also happening for prefect 3