Open anna-geller opened 2 years ago
Here's the (internal) commit that used that design: https://github.com/PrefectHQ/orion/pull/770/commits/8d6beda52d4d7c4458d56ca07cbeccce8327eaf4 and details about why this was not used in the alternatives considered section of https://github.com/PrefectHQ/orion/pull/770
Logger levels are a bit more complicated as they're generally controlled by the logging.yml
file and Prefect settings. We also don't want to overload the Python Logger
type. We could expose a factory still e.g. by having get_run_logger
work at the top-level but have it dynamically assign the correct logger name based on context for each call.
Here's the relevant section of the pull request:
Instead of calling a function in the task/flow to retrieve a logger, we considered having the logger available as a top-level import. This is in a similar style to https://github.com/Delgan/loguru
from prefect import flow, task, logger
@task
def foo():
logger.info("Hello from the 'foo' task")
@flow
def test_flow():
logger.info("Hello from the parent flow!")
foo()
test_flow()
Here, the context would be checked when the log is sent and the log would be routed to the correct logger. We opted not to do this as the logger
could behave unexpectedly if you attempt to attach handlers manually and a run_logger
function is easier to reason about.
Mostly, we did not want to prematurely optimize by making the logger more complicated than it needs to be.
Here's the relevant section of the pull request:
Importable logger
Instead of calling a function in the task/flow to retrieve a logger, we considered having the logger available as a top-level import. This is in a similar style to https://github.com/Delgan/loguru
from prefect import flow, task, logger @task def foo(): logger.info("Hello from the 'foo' task") @flow def test_flow(): logger.info("Hello from the parent flow!") foo() test_flow()
Here, the context would be checked when the log is sent, and the log would be routed to the correct logger. We opted not to do this as the
logger
could behave unexpectedly if you attempt to attach handlers manually and arun_logger
function is easier to reason about.
I've used Loguru as a 3rd party logging library which redirects existing logging to Loguru sink with various customized handlers attached to it. It is an amazing library which eased most of my logging configuration and logging needs. One of the major use cases was having the logs exported to an external API (In my case it was Grafana-Loki) during runtime where it acted as a plug-and-play solution.
My stack was a FastAPI + Uvicorn server running jobs where all logs needed to be streamed to Grafana-Loki. Loguru was my choice of logging which redirected all native Python logging, library logging towards Loguru's sink seamlessly. Grafana-Loki log stream was done with a custom handler which could customize the log format, shape and size that streamed to Loki. This meant that a user could stream logs in a string format on the terminal and choose to opt over a serialized log exporter to Loki if needed or a serialized log exporter to an offline file (Loguru's built in features) as required.
As far as the discussion thread above is concerned, integrating Loguru to Prefect makes a whole lot of sense, it would allow task/flow level loggers to propagate logs to Loguru sink with all existing configuration and allow non-prefect code to invoke logger from Loguru for their logging needs.
The way I usually configure/customize Loguru is at the start-up, move all existing loggers (or relevant loggers) to Loguru sink, attach required handlers. Since it is a one-time-job, it doesn't need extensive changes to any existing logging configuration on most of the projects.
I can lend a hand if needed to get this done, I use Prefect and would love to see Loguru working out.
PS: I am not able to view any commits/PR links mentioned above, it redirects to a page not found error on GitHub.
@cicdw perhaps we should add this at prefect.runtime.logger
?
I like that idea, yea!
First check
Prefect Version
2.x
Describe the current behavior
Currently, Prefect logger can only be used in an active flow/task run context
Users are requesting that the same logger can also be used when running things ad-hoc, from custom non-Prefect scripts and e.g. from tests: https://discourse.prefect.io/t/hybrid-logger-a-logger-that-works-both-inside-and-outsite-of-a-prefect-context/1559
Describe the proposed behavior
The original implementation of that was:
@madkinsz suggested we could bring that behavior back