Structured logging with lists & dicts

[NobleMathews]

Hey ! 👋 🙂

I've been through quite some "issues" with a similar topic (including which I have also participated in! ) but I thought perhaps a new thread is worth it, instead of reviving a closed one. 😅

hashtag Logging Lists & Dicts

Previously, you helped me get to [this]( point:

import sys

from loguru import logger

def formatter(record):
    base_format = "{time} {level} {name} {message} " + " " * 10
    base = base_format.format_map(record)
    lines = str(record["extra"].get("data", "")).splitlines()
    indent = "\n" + " " * len(base)
    reformatted = base + indent.join(lines)
    record["extra"]["reformatted"] = reformatted
    return "{extra[reformatted]}\n{exception}"

logger.remove()
logger.add(sys.stderr, format=formatter)

data = """---------------- Request ----------------
Headers       :   {"Accept": "*/*",
                   "Accept-Encoding": "gzip, deflate",
                   "Connection": "keep-alive",
                   "Content-Length": "20",
                   "User-Agent": "python-requests/2.27.1",
                   "cookie": "foo=bar; bar=baz",
                   "x-pretty-print": "2"}
URL           :   
Method        :   POST
Body          :   {"foo": "bar"}"""

logger.info("Default message")
logger.bind(data=data).info("Message with data")

However, in the above example, I was logging a str, and instead, I'd like to log a dict:

{
        "Headers": {
            "Accept": "*/*",
            "Accept-Encoding": "gzip, deflate",
            "Connection": "keep-alive",
            "Content-Length": "20",
            "User-Agent": "python-requests/2.27.1",
            "cookie": "foo=bar; bar=baz",
            "x-pretty-print": "2",
        },
        "URL": "
        "Method": "POST",
        "Body": {
            "foo": "bar",
        },
    }

1. Is there a newer, better way of the above solution? Have things changed perhaps? 👀
2. How could I make it so that I can "auto-format" the output for lists or dicts?

hashtag Logging with indents

Wrt OP's original query, I'd like to also have indentation, but slightly different.

This was the proposed solution:

logger_indentation = ContextVar("logger_indentation", default=0)

def indent_logs(indent_size):
    val = logger_indentation.get()
    logger_indentation.set(val + indent_size)
    yield
    logger_indentation.set(val)

def patcher(record):
    indentation = logger_indentation.get()
    record.update(message=" " * indentation + record["message"])

And in my use-case, I am doing this:

  logger.info(f"Parameters:")
    for param_info in param_info_list:
      with indent_logs(4):
        logger.info(param_info)

With output:

2023-12-04 15:31:27.057 | INFO     | actions_wrapper:action_wrapper:129 - Parameters:
2023-12-04 15:31:27.057 | INFO     | actions_wrapper:action_wrapper:132 -     element = "some_str"
2023-12-04 15:31:27.058 | INFO     | actions_wrapper:action_wrapper:132 -     element_index = 0 (Default)

But, what I'd preferably like to have is one log statement:

2023-12-04 15:31:27.057 | INFO     | actions_wrapper:action_wrapper:129 - Parameters:
                                                                             element         = "some_str"
                                                                             element_index   = 0 (Default)

And preferably not having to use a context manager - maybe something similar to using bind()? 🤔

[mayil-ai[bot]]

Possible Solution

To address the issue described, you can implement the following changes:

• Modify the formatter function to check if the message is a dict or list and format it using json.dumps with indentation if it is.
• Implement a patcher function that modifies the record to include the indentation level from logger_indentation and applies it to the message.
• Use logger.patch(patcher) to apply the indentation to all messages.
• Replace the context manager with indent_logs(4): with logger.bind(indent=4) to set the indentation level using bind() instead of a context manager.
• Update the patcher function to read the indentation level from record["extra"].get("indent", 0) and apply it to the message.

Here is a sample implementation:

import json
from loguru import logger
from contextvars import ContextVar

logger_indentation = ContextVar("logger_indentation", default=0)

def formatter(record):
    message = record["message"]
    if isinstance(message, (dict, list)):
        # Format message as JSON with indentation
        message = json.dumps(message, indent=4)
    record["message"] = message
    return "{time} - {level} - {message}\n"

def patcher(record):
    base_indent = logger_indentation.get()
    indent = record["extra"].get("indent", 0)
    total_indent = ' ' * (base_indent + indent)
    record["message"] = total_indent + record["message"].replace('\n', '\n' + total_indent)

logger.configure(handlers=[{"sink": sys.stderr, "format": formatter}])
logger.patch(patcher)

# Usage example
logger.bind(indent=4).info("Indented message")

This code snippet assumes that the formatter function is used to format log messages and that the patcher function is used to apply indentation. The logger_indentation context variable is used to keep track of the current indentation level, and the bind() method is used to set the indentation level for specific log messages.

Code snippets to check

loguru → _logger.py

This snippet contains the 'bind' method which is relevant to the issue as the user is looking for a way to log structured data like dicts without using a context manager. https://github.com/Mayil-AI/loguru/blob/871de74acb2382fb3982d3fb8e224e4a82de9073/loguru/_logger.py#L1385-L2086 This snippet describes the 'add' method and its parameters, which is relevant for understanding how to configure the logger to auto-format lists or dicts. https://github.com/Mayil-AI/loguru/blob/871de74acb2382fb3982d3fb8e224e4a82de9073/loguru/_logger.py#L253-L2094 This snippet contains the 'opt' method which could be relevant for the issue as it allows customization of the logging call, which may be useful for the desired indentation and formatting. https://github.com/Mayil-AI/loguru/blob/871de74acb2382fb3982d3fb8e224e4a82de9073/loguru/_logger.py#L1297-L2094

Debugging information

Clarification Questions

NONE

• Breakdown of AI Costs:

  search_embedding: 1.3e-07

  snippet_processor: 0.13788000000000003

  issue_star_creation: 0.03249

  issue_star_solver: 0.14106000000000002

  bouncer: 0.02975