Structured logs

[vikramsubramanian]

I've been through quite some "issues" with a similar topic (including which I have also participated in! ) but I thought perhaps a new thread is worth it, instead of reviving a closed one. 😅

hashtag Logging Lists & Dicts

Previously, you helped me get to [this]( point:

import sys

from loguru import logger

def formatter(record): base_format = "{time} {level} {name} {message} " + " " 10 base = base_format.format_map(record) lines = str(record["extra"].get("data", "")).splitlines() indent = "\n" + " " len(base) reformatted = base + indent.join(lines) record["extra"]["reformatted"] = reformatted return "{extra[reformatted]}\n{exception}"

logger.remove() logger.add(sys.stderr, format=formatter)

data = """---------------- Request ---------------- Headers : {"Accept": "/", "Accept-Encoding": "gzip, deflate", "Connection": "keep-alive", "Content-Length": "20", "User-Agent": "python-requests/2.27.1", "cookie": "foo=bar; bar=baz", "x-pretty-print": "2"} URL : Method : POST Body : {"foo": "bar"}"""

logger.info("Default message") logger.bind(data=data).info("Message with data") However, in the above example, I was logging a str, and instead, I'd like to log a dict:

{ "Headers": { "Accept": "/", "Accept-Encoding": "gzip, deflate", "Connection": "keep-alive", "Content-Length": "20", "User-Agent": "python-requests/2.27.1", "cookie": "foo=bar; bar=baz", "x-pretty-print": "2", }, "URL": " "Method": "POST", "Body": { "foo": "bar", }, } Is there a newer, better way of the above solution? Have things changed perhaps? 👀 How could I make it so that I can "auto-format" the output for lists or dicts? hashtag Logging with indents

Wrt OP's original query, I'd like to also have indentation, but slightly different.

This was the proposed solution:

logger_indentation = ContextVar("logger_indentation", default=0)

def indent_logs(indent_size): val = logger_indentation.get() logger_indentation.set(val + indent_size) yield logger_indentation.set(val)

def patcher(record): indentation = logger_indentation.get() record.update(message=" " * indentation + record["message"]) And in my use-case, I am doing this:

logger.info(f"Parameters:") for param_info in param_info_list: with indent_logs(4): logger.info(param_info) With output:

2023-12-04 15:31:27.057 | INFO | actions_wrapper:action_wrapper:129 - Parameters: 2023-12-04 15:31:27.057 | INFO | actions_wrapper:action_wrapper:132 - element = "some_str" 2023-12-04 15:31:27.058 | INFO | actions_wrapper:action_wrapper:132 - element_index = 0 (Default) But, what I'd preferably like to have is one log statement:

2023-12-04 15:31:27.057 | INFO | actions_wrapper:action_wrapper:129 - Parameters: element = "some_str" element_index = 0 (Default) And preferably not having to use a context manager - maybe something similar to using bind()? 🤔

[mayil-ai[bot]]

Possible Solution

• Update the formatter function to handle dictionary data by serializing it to JSON if the record["extra"].get("data") is a dictionary.
• Add a patcher function that modifies the record["extra"] to include indentation based on the logger_indentation ContextVar.
• Use logger.patch(patcher) to apply the indentation to all log messages.
• Replace direct calls to logger.info() with a wrapper function that increments the logger_indentation before logging and decrements it afterward.

from loguru import logger
from contextvars import ContextVar
import json

logger_indentation = ContextVar("logger_indentation", default=0)

def formatter(record):
    format_string = "{time} - {level} - {message}"
    if "data" in record["extra"]:
        record["extra"]["data"] = json.dumps(record["extra"]["data"], indent=4)
    return format_string

def patcher(record):
    current_indentation = logger_indentation.get()
    prefix = " " * current_indentation
    record["message"] = prefix + record["message"]

def indent_logger(indent_size):
    current_indentation = logger_indentation.get()
    logger_indentation.set(current_indentation + indent_size)
    try:
        yield
    finally:
        current_indentation = logger_indentation.get()
        logger_indentation.set(current_indentation - indent_size)

logger.remove()
logger.add(sys.stderr, format=formatter)
logger.patch(patcher)

# Usage of logger with dictionary data
log_data = {
    "Headers": { ... },
    "URL": ...,
    "Method": "POST",
    "Body": { ... },
}
logger.bind(data=log_data).info("Message with data")

# Usage of logger with automatic indentation
with indent_logger(4):
    logger.info("Parameters:")
    with indent_logger(4):
        logger.info("element = 'some_str'")
        logger.info("element_index = 0 (Default)")

• Ensure that the formatter function is set correctly with logger.add(sys.stderr, format=formatter).
• Use logger.patch(patcher) to apply the patcher function globally.
• Use the indent_logger context manager to handle indentation around log calls.

Code snippets to check

loguru → _logger.py

This snippet contains the 'add' method of the logger which is relevant for configuring the logger to handle structured logs, including the 'format' parameter that can be used to define how to format dicts and lists. https://github.com/Mayil-AI/loguru/blob/871de74acb2382fb3982d3fb8e224e4a82de9073/loguru/_logger.py#L253-L2094 This snippet is relevant because it shows the 'bind' method which can be used to attach additional context to log messages, which could be useful for structured logging of dicts. https://github.com/Mayil-AI/loguru/blob/871de74acb2382fb3982d3fb8e224e4a82de9073/loguru/_logger.py#L1385-L2086 This snippet is relevant as it shows how log records are constructed and formatted, which is essential for understanding how to auto-format output for lists or dicts in structured logs. https://github.com/Mayil-AI/loguru/blob/871de74acb2382fb3982d3fb8e224e4a82de9073/loguru/_logger.py#L1981-L2033

Debugging information

Clarification Questions

NONE

• Breakdown of AI Costs:

  search_embedding: 1.3e-07

  snippet_processor: 0.12919000000000003

  issue_star_creation: 0.030539999999999998

  issue_star_solver: 0.13477

  bouncer: 0.028560000000000002