Add a "hybrid" Prefect logger that can be used both from run context and outside of it

[anna-geller]

First check

• [X] I added a descriptive title to this issue.
• [X] I used the GitHub search to find a similar request and didn't find it.
• [X] I searched the Prefect documentation for this feature.

Prefect Version

2.x

Describe the current behavior

Currently, Prefect logger can only be used in an active flow/task run context

Users are requesting that the same logger can also be used when running things ad-hoc, from custom non-Prefect scripts and e.g. from tests: https://discourse.prefect.io/t/hybrid-logger-a-logger-that-works-both-inside-and-outsite-of-a-prefect-context/1559

Describe the proposed behavior

The original implementation of that was:

from prefect import logger

logger.info("Logging from anywhere")

@madkinsz suggested we could bring that behavior back

[zanieb]

Here's the (internal) commit that used that design: https://github.com/PrefectHQ/orion/pull/770/commits/8d6beda52d4d7c4458d56ca07cbeccce8327eaf4 and details about why this was not used in the alternatives considered section of https://github.com/PrefectHQ/orion/pull/770

[zanieb]

Logger levels are a bit more complicated as they're generally controlled by the logging.yml file and Prefect settings. We also don't want to overload the Python Logger type. We could expose a factory still e.g. by having get_run_logger work at the top-level but have it dynamically assign the correct logger name based on context for each call.

[zanieb]

Here's the relevant section of the pull request:

Importable logger

Instead of calling a function in the task/flow to retrieve a logger, we considered having the logger available as a top-level import. This is in a similar style to https://github.com/Delgan/loguru

from prefect import flow, task, logger

@task
def foo():
    logger.info("Hello from the 'foo' task")

@flow
def test_flow():
    logger.info("Hello from the parent flow!")
    foo()

test_flow()

Here, the context would be checked when the log is sent and the log would be routed to the correct logger. We opted not to do this as the logger could behave unexpectedly if you attempt to attach handlers manually and a run_logger function is easier to reason about.

[zanieb]

Mostly, we did not want to prematurely optimize by making the logger more complicated than it needs to be.

[amruthvvkp]

Here's the relevant section of the pull request:

Importable logger

Instead of calling a function in the task/flow to retrieve a logger, we considered having the logger available as a top-level import. This is in a similar style to https://github.com/Delgan/loguru

from prefect import flow, task, logger

@task
def foo():
    logger.info("Hello from the 'foo' task")

@flow
def test_flow():
    logger.info("Hello from the parent flow!")
    foo()

test_flow()

Here, the context would be checked when the log is sent, and the log would be routed to the correct logger. We opted not to do this as the logger could behave unexpectedly if you attempt to attach handlers manually and a run_logger function is easier to reason about.

I've used Loguru as a 3rd party logging library which redirects existing logging to Loguru sink with various customized handlers attached to it. It is an amazing library which eased most of my logging configuration and logging needs. One of the major use cases was having the logs exported to an external API (In my case it was Grafana-Loki) during runtime where it acted as a plug-and-play solution.

My stack was a FastAPI + Uvicorn server running jobs where all logs needed to be streamed to Grafana-Loki. Loguru was my choice of logging which redirected all native Python logging, library logging towards Loguru's sink seamlessly. Grafana-Loki log stream was done with a custom handler which could customize the log format, shape and size that streamed to Loki. This meant that a user could stream logs in a string format on the terminal and choose to opt over a serialized log exporter to Loki if needed or a serialized log exporter to an offline file (Loguru's built in features) as required.

As far as the discussion thread above is concerned, integrating Loguru to Prefect makes a whole lot of sense, it would allow task/flow level loggers to propagate logs to Loguru sink with all existing configuration and allow non-prefect code to invoke logger from Loguru for their logging needs.

The way I usually configure/customize Loguru is at the start-up, move all existing loggers (or relevant loggers) to Loguru sink, attach required handlers. Since it is a one-time-job, it doesn't need extensive changes to any existing logging configuration on most of the projects.

I can lend a hand if needed to get this done, I use Prefect and would love to see Loguru working out.

PS: I am not able to view any commits/PR links mentioned above, it redirects to a page not found error on GitHub.

[zanieb]

@cicdw perhaps we should add this at prefect.runtime.logger?

[cicdw]

I like that idea, yea!