PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.89k stars 1.56k forks source link

Logging not clear what I should be doing to get expected results #15370

Open j-carson opened 1 week ago

j-carson commented 1 week ago

Bug summary

I am converting a command line data pipeline to a prefect flow and am trying to get logging to work as expected. I was discussing this with Chris White here: https://prefect-community.slack.com/archives/CL09KU1K7/p1726094132119519 But the source code is a bit complicated and he asked me to put it in a ticket instead. The code is here: https://github.com/j-carson/prefect_log_question

Version info (prefect version output)

Version:             3.0.1
API version:         0.8.4
Python version:      3.10.14
Git commit:          c6b2ffe1
Built:               Fri, Sep 6, 2024 10:05 AM
OS/Arch:             linux/x86_64
Profile:             cloud
Server type:         cloud
Pydantic version:    2.8.2

Additional context

I put the context in the comments in the flow in the linked repository

j-carson commented 1 day ago

I've stared at this a bit, I think what needs to happen is here:

logging/configuration.py line 104:

            if logger.level == logging.NOTSET:
                logger.setLevel(extra_config.level)

should just be

handler.setLevel(extra_config.level)
  1. No if-check because logger.level is always logging.NOTSET because configuration happens before my code has a chance to run
  2. Set the level only on prefect's handler, not on the logger my code wants to configure differently