Structured logging with lists & dicts (hashtag1040)

[vikramsubramanian]

Hey ! 👋 🙂

I've been through quite some "issues" with a similar topic (including which I have also participated in! ) but I thought perhaps a new thread is worth it, instead of reviving a closed one. 😅

hashtag Logging Lists & Dicts

Previously, you helped me get to [this]( point:

import sys

from loguru import logger

def formatter(record):
    base_format = "{time} {level} {name} {message} " + " " * 10
    base = base_format.format_map(record)
    lines = str(record["extra"].get("data", "")).splitlines()
    indent = "\n" + " " * len(base)
    reformatted = base + indent.join(lines)
    record["extra"]["reformatted"] = reformatted
    return "{extra[reformatted]}\n{exception}"

logger.remove()
logger.add(sys.stderr, format=formatter)

data = """---------------- Request ----------------
Headers       :   {"Accept": "*/*",
                   "Accept-Encoding": "gzip, deflate",
                   "Connection": "keep-alive",
                   "Content-Length": "20",
                   "User-Agent": "python-requests/2.27.1",
                   "cookie": "foo=bar; bar=baz",
                   "x-pretty-print": "2"}
URL           :   
Method        :   POST
Body          :   {"foo": "bar"}"""

logger.info("Default message")
logger.bind(data=data).info("Message with data")

However, in the above example, I was logging a str, and instead, I'd like to log a dict:

{
        "Headers": {
            "Accept": "*/*",
            "Accept-Encoding": "gzip, deflate",
            "Connection": "keep-alive",
            "Content-Length": "20",
            "User-Agent": "python-requests/2.27.1",
            "cookie": "foo=bar; bar=baz",
            "x-pretty-print": "2",
        },
        "URL": "
        "Method": "POST",
        "Body": {
            "foo": "bar",
        },
    }

1. Is there a newer, better way of the above solution? Have things changed perhaps? 👀
2. How could I make it so that I can "auto-format" the output for lists or dicts?

hashtag Logging with indents

Wrt OP's original query, I'd like to also have indentation, but slightly different.

This was the proposed solution:

logger_indentation = ContextVar("logger_indentation", default=0)

def indent_logs(indent_size):
    val = logger_indentation.get()
    logger_indentation.set(val + indent_size)
    yield
    logger_indentation.set(val)

def patcher(record):
    indentation = logger_indentation.get()
    record.update(message=" " * indentation + record["message"])

And in my use-case, I am doing this:

  logger.info(f"Parameters:")
    for param_info in param_info_list:
      with indent_logs(4):
        logger.info(param_info)

With output:

2023-12-04 15:31:27.057 | INFO     | actions_wrapper:action_wrapper:129 - Parameters:
2023-12-04 15:31:27.057 | INFO     | actions_wrapper:action_wrapper:132 -     element = "some_str"
2023-12-04 15:31:27.058 | INFO     | actions_wrapper:action_wrapper:132 -     element_index = 0 (Default)

But, what I'd preferably like to have is one log statement:

2023-12-04 15:31:27.057 | INFO     | actions_wrapper:action_wrapper:129 - Parameters:
                                                                             element         = "some_str"
                                                                             element_index   = 0 (Default)

And preferably not having to use a context manager - maybe something similar to using bind()? 🤔 )

[mayil-ai[bot]]

Possible Solution

• Configure the logger to use a custom formatter that handles JSON serialization with indentation:

  import json
  from loguru import logger
  
  def json_formatter(record):
    record["extra"]["data"] = json.dumps(record["extra"].get("data", {}), indent=4)
    return "{time} - {level} - {message} - {extra[data]}\n"
  
  logger.remove()
  logger.add(sys.stderr, format=json_formatter)

• Use logger.bind() to attach the dict data you want to log:

  logger.bind(data={"key": "value"}).info("Log message")

• If you want to apply indentation globally without using bind() every time, you can patch the logger:

  from loguru import logger
  
  def patcher(record):
    if isinstance(record["extra"].get("data"), dict):
        record["extra"]["data"] = json.dumps(record["extra"]["data"], indent=4)
  
  logger.configure(patcher=patcher)

• Then, log your dict using the extra parameter:

  logger.info("Log message", extra={"data": {"key": "value"}})

• Ensure that the extra parameter is not used for other purposes in your logging calls, as it is now reserved for dict data that needs to be formatted.

Code snippets to check

loguru → _logger.py

This snippet contains the 'add' method of the logger which is relevant for configuring the logging format and could be modified to handle dict and list formatting. https://github.com/Mayil-AI/loguru/blob/871de74acb2382fb3982d3fb8e224e4a82de9073/loguru/_logger.py#L253-L2094 This snippet contains the 'bind' method which is relevant for adding structured data to logs and could be used to improve the logging of dicts and lists. https://github.com/Mayil-AI/loguru/blob/871de74acb2382fb3982d3fb8e224e4a82de9073/loguru/_logger.py#L1385-L2086 This snippet shows how the log record is constructed and formatted, which is relevant for understanding how to auto-format lists and dicts in the log output. https://github.com/Mayil-AI/loguru/blob/871de74acb2382fb3982d3fb8e224e4a82de9073/loguru/_logger.py#L1981-L2033

loguru → _handler.py

This snippet includes the handler initialization which is relevant for understanding how the formatter and colorizer are set up, which could be modified to handle dict and list formatting. https://github.com/Mayil-AI/loguru/blob/871de74acb2382fb3982d3fb8e224e4a82de9073/loguru/_handler.py#L32-L108