Suggestion: Parse log files à la json.tool

[ROpdebee]

I'm currently using a logging setup along the lines of the following:

logger.add('output.log', serialize=True, level='TRACE')

which dumps a whole load of JSON-serialized log messages to the output.log file. I'm using this in order to easily review logging events in case of a mishap. It would be awesome if loguru were to provide a CLI tool to easily parse such log files, capable of using both Logger.parse as well as direct JSON deserialization.

As a reference, running python -m json.tool on a JSON file pretty prints the JSON content to the terminal. I think a "loguru.tool" should do something similar.

Concretely, I propose such a tool should have the following features, in order of desirability:

• [ ] The ability to deserialize a file with JSON records as produced using a logging setup as shown above.
• [ ] Nicely structured, colorized output.
• [ ] The ability to parse using a custom regex, like Logger.parse
• [ ] The ability to --select a number of fields, e.g. --select=level.name,message would select the log level and message
• [ ] The ability to --filter based on a condition on these fields, e.g. --filter="module==my_module" would only display messages that originate from the my_module module. Perhaps the last two would only be applicable using the JSON-formatted records or a regex using named capture groups.

This would make it easier for developers to track down only the log messages that they're actually interested in and view them in a user-friendly manner, without having to write custom scripts or firing up a REPL all the time, making Python logging truly stupidly simple in every step of the process.

Great project btw, I'm really liking the easy setup compared to the woes of the standard logging module! :heart:

[ROpdebee]

I've been playing around with the idea a bit and quickly hacked together a PoC derived from Python's own json.tool: https://gist.github.com/ROpdebee/fa86464ee6410078f9ae50c35f997ad4 Can do stuff like

python -m loguru.tool output.log --select=function --select=message --filter="level.no>=20" --filter="module!=main"

"It works", but there's plenty of ways to make it...not work, let's say. There's very little exception handling and input checking, the parsing and handling of filter expressions is atrocious and there's no type safety at all, really. It serves only as an example and potential starting point.

I could probably continue working on it and turn it into a PR, if you're interested.

[Delgan]

Hey @ROpdebee , thanks for your suggestion and kind words! :smile:

This is an interesting idea, however this is in my opinion out of scope for this library. I try to embrace the Unix philosophy "do one thing and do it well". In this sense, I think the .parse() method was maybe a mistake. I would prefer loguru to be limited to a logging and not parsing library. As you have found creating a PoC, this is a tricky topic!

Why not create an independent script or library? Actually, I think your tool could be even more powerful that way. Your are not limited to loguru output, you could for example parse logs from structlog as they are easy to parse too.

[ROpdebee]

Why not create an independent script or library? Actually, I think your tool could be even more powerful that way. Your are not limited to loguru output, you could for example parse logs from structlog as they are easy to parse too.

Hah, maybe in another life! The parsing itself is pretty straightforward, as shown in the gist, but the filtering would lead to many, many headaches I think. Especially because there's many different axes to consider:

• Types, especially container types would require a lot of work to be implemented reliably
• Predicates: simple comparisons are fine, but what about an includes predicate, regex matching, sequence length, ...
• Context-sensitive filtering: "Give me the last 5 debug lines preceding an error message from this module."
• ...

All-in-all, these would be simpler to implement programmatically rather than feeding them to a CLI tool using its own syntax. It's probably even easier to learn a JSON query language like JSONiq and use that instead! There's a reason this was at the bottom of the checklist :wink: And I'm really bad at making terminal output look good, so there's that too...

I'll keep it in the back of my head if I'm in need of a project in the future, thanks for the feedback!

[Delgan]

Thanks to you for sharing your thoughts!

Also, worth noting services like Loggly and Sentry which are used to aggregate and post-process logs of all kinds.