Optimize Log initialization for parallel execution and prefect integration

[windiana42]

Log initialization is a tricky subject. The logging library does not provide satisfactory guidance for various software engineering usecases. Thus everyone hacks around this gap and those hacks can interact badly.

Ideal world:

• emitting log messages is decoupled from log handler initialization/configuration
• a simple import logging; logger=logging.getLogger(__name__) should do on the log message emitting side

Late initialization:

• since it is hard to make sure logging is initialized during import phase in python, it may be discouraged to get a logger objects on module level
• there are three techniques to ensure late initialization of logger after import phase yields the correct result:
  1. Loggers may be created but not used during module declaration time (import phase); Logger initialization updates existing loggers to its desired configuration (i.e. disable_existing_loggers=False).
  2. Loggers as class members class X; def __init__(self):; self.logger = logging.getLogger(X.__class__.__name__)
  3. Loggers in each function def function_name():; logger = logging.getLogger(__name__+"-function_name")

Problems for log initialization that need solving:

• [ ] The library prefect initializes logging during import prefect. It captures console logs which is bad for debugging. It offers a handler named debug but it is unclear how to activate it. Even calling pipedag structlog.setup_logging() afterwards kills emitted messages by non-prefect initialized loggers. logging.getLogger("root").handlers = [] before calling structlog.setup_logging() helps to some degree. But prefect logs are still not formatted as nice as for non-prefect loggers.
• [ ] The prefect library asks tasks to initialize a prefect specific logger in order to make logs available per task. With a parallel multi-node executor, can we avoid this technique and still associate emitted log messages with a specific task in case we wanted to offer similar functionalities like only outputting logs of failing tasks?
• [ ] If we can avoid special logger initialization for logging task specific messages, can we also forward those messages to prefect in case we use it as a backend?
• [ ] prefect modifies root logger and derived loggers not only on import prefect but also on starting flow. So it must be ensured that existing and new loggers work correctly also after executing flow.

[windiana42]

@NMAC427 I am currently not sure how much effort we want to invest in prefect. It is used in some projects today. But I couldn't find a way to connect runs which are not triggered by an explicit prefect process to the run-database which the UI would pick up. In addition, prefect 2 seems to go in a direction (async-await) which is different from pipedag which means interests for features to come might not be aligned. You can see this in the radar diagram of displaying tasks which is super ugly in case the data pipeline in-deed is rather linear with some branching out happening in each pipeline stage: https://www.prefect.io/guide/blog/introducing-radar/ However, prefect 2.10 supports more dag like diagrams now again: https://docs.prefect.io/2.10.17/

So the investigation of alternative DAG orchestration UIs might influence the importance of this issue. If it is easy to fix prefect compatibility, it might be a no-brainer to do it since we already support it as a backend.