Structured logging with lists & dicts (hashtag1040)

[vikramsubramanian]

Hey ! 👋 🙂

I've been through quite some "issues" with a similar topic (including which I have also participated in! ) but I thought perhaps a new thread is worth it, instead of reviving a closed one. 😅

hashtag Logging Lists & Dicts

Previously, you helped me get to [this]( point:

import sys

from loguru import logger

def formatter(record):
    base_format = "{time} {level} {name} {message} " + " " * 10
    base = base_format.format_map(record)
    lines = str(record["extra"].get("data", "")).splitlines()
    indent = "\n" + " " * len(base)
    reformatted = base + indent.join(lines)
    record["extra"]["reformatted"] = reformatted
    return "{extra[reformatted]}\n{exception}"

logger.remove()
logger.add(sys.stderr, format=formatter)

data = """---------------- Request ----------------
Headers       :   {"Accept": "*/*",
                   "Accept-Encoding": "gzip, deflate",
                   "Connection": "keep-alive",
                   "Content-Length": "20",
                   "User-Agent": "python-requests/2.27.1",
                   "cookie": "foo=bar; bar=baz",
                   "x-pretty-print": "2"}
URL           :   
Method        :   POST
Body          :   {"foo": "bar"}"""

logger.info("Default message")
logger.bind(data=data).info("Message with data")

However, in the above example, I was logging a str, and instead, I'd like to log a dict:

{
        "Headers": {
            "Accept": "*/*",
            "Accept-Encoding": "gzip, deflate",
            "Connection": "keep-alive",
            "Content-Length": "20",
            "User-Agent": "python-requests/2.27.1",
            "cookie": "foo=bar; bar=baz",
            "x-pretty-print": "2",
        },
        "URL": "
        "Method": "POST",
        "Body": {
            "foo": "bar",
        },
    }

1. Is there a newer, better way of the above solution? Have things changed perhaps? 👀
2. How could I make it so that I can "auto-format" the output for lists or dicts?

hashtag Logging with indents

Wrt OP's original query, I'd like to also have indentation, but slightly different.

This was the proposed solution:

logger_indentation = ContextVar("logger_indentation", default=0)

def indent_logs(indent_size):
    val = logger_indentation.get()
    logger_indentation.set(val + indent_size)
    yield
    logger_indentation.set(val)

def patcher(record):
    indentation = logger_indentation.get()
    record.update(message=" " * indentation + record["message"])

And in my use-case, I am doing this:

  logger.info(f"Parameters:")
    for param_info in param_info_list:
      with indent_logs(4):
        logger.info(param_info)

With output:

2023-12-04 15:31:27.057 | INFO     | actions_wrapper:action_wrapper:129 - Parameters:
2023-12-04 15:31:27.057 | INFO     | actions_wrapper:action_wrapper:132 -     element = "some_str"
2023-12-04 15:31:27.058 | INFO     | actions_wrapper:action_wrapper:132 -     element_index = 0 (Default)

But, what I'd preferably like to have is one log statement:

2023-12-04 15:31:27.057 | INFO     | actions_wrapper:action_wrapper:129 - Parameters:
                                                                             element         = "some_str"
                                                                             element_index   = 0 (Default)

And preferably not having to use a context manager - maybe something similar to using bind()? 🤔 )

[mayil-ai[bot]]

Possible Solution

• Create a new formatter function that can handle JSON serialization of dicts and lists for logging.
• Modify the patcher function to apply indentation based on the logger_indentation level.
• Use the logger.bind() method to attach structured data to logs without a context manager.
• Configure the logger to use the new formatter and patcher functions.

# Define a formatter function that serializes dicts and lists to JSON
def formatter(record):
    message = record["message"]
    if isinstance(message, (dict, list)):
        # Serialize the message to JSON format
        record["message"] = json.dumps(message, indent=logger_indentation.get(default=0))
    return "{time} - {level} - {message}\n"

# Define a patcher function that applies indentation
def patcher(record):
    indentation = ' ' * logger_indentation.get(default=0)
    record["message"] = indentation + record["message"]

# Configure the logger to use the new formatter and patcher
logger.configure(handlers=[{"sink": sys.stderr, "format": formatter, "patch": patcher}])

# Example usage of logger with bound data
logger.bind(data=some_dict).info("Message with data")

• Ensure that logger_indentation is a ContextVar that holds the current indentation level.
• The formatter function should check if the message is a dict or list and serialize it to JSON with indentation.
• The patcher function should prepend spaces to the message based on the current indentation level.
• Configure the logger with the new formatter and patcher to apply these changes to all logged messages.
• Use logger.bind() to attach additional structured data to the log message without using a context manager.

Code snippets to check

loguru → _logger.py

This snippet contains the 'add' method which is used to configure the logger, including formatting and serialization options, which are relevant to the issue of logging lists and dicts. https://github.com/Mayil-SB-Internal-Prod/loguru-II/blob/dcf42d962567723c0195bb8b2bb6fd764ecf41ae/loguru/_logger.py#L253-L2094 This snippet contains the 'bind' method which is used to bind extra attributes to the logger, which could be relevant for auto-formatting lists and dicts. https://github.com/Mayil-SB-Internal-Prod/loguru-II/blob/dcf42d962567723c0195bb8b2bb6fd764ecf41ae/loguru/_logger.py#L1385-L2086 This snippet contains the logging logic, including handling of 'extra' dict and message formatting, which is relevant to the issue of logging structured data like lists and dicts. https://github.com/Mayil-SB-Internal-Prod/loguru-II/blob/dcf42d962567723c0195bb8b2bb6fd764ecf41ae/loguru/_logger.py#L1981-L2033

loguru → _handler.py

This snippet contains the Handler class initialization, which includes formatter and colorize options that could be relevant for structured logging. https://github.com/Mayil-SB-Internal-Prod/loguru-II/blob/dcf42d962567723c0195bb8b2bb6fd764ecf41ae/loguru/_handler.py#L32-L108

loguru → _better_exceptions.py

This snippet contains the ExceptionFormatter class which might be relevant for customizing the formatting of exceptions, which could be part of structured logging. https://github.com/Mayil-SB-Internal-Prod/loguru-II/blob/dcf42d962567723c0195bb8b2bb6fd764ecf41ae/loguru/_better_exceptions.py#L122-L190