feature-request: Support for logfmt pattern layout

[apollo13]

It would be nice if loki-logback-appender supported logfmt layout by default (https://brandur.org/logfmt). Compared to the json output it has the upside of being more readable for humans (imo) and is supported by loki natively as well: https://grafana.com/docs/loki/latest/query/log_queries/#logfmt

[nehaev]

Hi @apollo13, thanks for reporting this!

Isn't it possible to use the existing pattern layout for logfmt-flavoured logs? E.g.,

<pattern>app=my-app host=${HOSTNAME} level=%level message=%msg</pattern>

[apollo13]

Hi @nehaev, thank you for the rapid response and the great question. While logfmt seems exceedingly simple on the surface, if your values have a space you need to escape them like thismsg="my message with spaces" and also if they then contain a " you'd have to escape that via \".

One could probably argue that this layout shouldn't be part of this package but then the same could probably be said for the json layout. On the upside if logfmt where to included it could mostly share the config with the json layout (which mdc keys to include etc) making it easy to switch between those two.

Does that make sense?

[nehaev]

Well, the reason why json layout was introduced in Loki4j is that it allows to customize and deliver the structured data to Loki. You can think of it as a carrier format for structures. So if you want your structured logs to be delivered to Loki as precise as possible, you just use json layout. Having logfmt as another carrier format does not add up anything really, json already covers this use case.

Human-readability is not about delivery, it's about representation. Even if you have your data delivered to Loki as json, it doesn't mean you have to read this raw json all the time. You can represent your structured data as plain text or logfmt on Grafana side using transformations available.

[apollo13]

You can represent your structured data as plain text or logfmt on Grafana side using transformations available.

Sure, but it requires quite a bit more processing and advanced logql queries (if logql even supports reencoding). Either way, I am fine if you don't want to support logfmt, I can write my own layout for our codebases as well, would have been nice though to give the user a choice between the two loki supported formats (json and logfmt, the others like regex or pattern don't really need anything)

[nehaev]

If you need a generic solution, then a custom layout is probably a way to go.

However, if what you need is to have logfmt configured for a particular project/app, you can save a lot of time and efforts by simply using a plain text pattern as I suggested in the first comment, and things like %prefix and %replace.