logfmt pipeline stage is polluting error logs

[shayycisco]

When using the logfmt stage and the log line isn't in this format you get bunch of errors in promtail:

level=error ts=2022-05-02T07:01:26.4715226Z caller=logfmt.go:139 component=file_pipeline component=stage type=logfmt msg="failed to decode logfmt" err="logfmt syntax error at pos 45 on line 1: unexpected '\"'"
level=error ts=2022-05-02T07:01:26.4721745Z caller=logfmt.go:139 component=file_pipeline component=stage type=logfmt msg="failed to decode logfmt" err="logfmt syntax error at pos 45 on line 1: unexpected '\"'"
level=error ts=2022-05-02T07:01:26.4725254Z caller=logfmt.go:139 component=file_pipeline component=stage type=logfmt msg="failed to decode logfmt" err="logfmt syntax error at pos 45 on line 1: unexpected '\"'"
level=error ts=2022-05-02T07:01:26.4729279Z caller=logfmt.go:139 component=file_pipeline component=stage type=logfmt msg="failed to decode logfmt" err="logfmt syntax error at pos 2 on line 1: unexpected '\"'"
level=error ts=2022-05-02T07:01:26.4730569Z caller=logfmt.go:139 component=file_pipeline component=stage type=logfmt msg="failed to decode logfmt" err="logfmt syntax error at pos 45 on line 1: unexpected '\"'"

Is it fair to say that these logs belong to debug? same as the unmarshall of the json stage

[slinkytoad]

This is almost a year old at this point, but did you ever sort this out?

[frittentheke]

I just ran into issues with the logfmt stage complaining about parsing errors ...

1. The error originates from https://github.com/grafana/loki/blob/6d2b3e8b76ef11a0f1308bd91ccb1dc071d17b55/clients/pkg/logentry/stages/logfmt.go#L139
2. Unfortunately there is no info about which file the parsing error belongs to or which string caused the error.
3. If you look further in logfmt.go you see that even when enabling debug logging to get more information, the error happens before any of the debug logging actually happens in https://github.com/grafana/loki/blob/6d2b3e8b76ef11a0f1308bd91ccb1dc071d17b55/clients/pkg/logentry/stages/logfmt.go#L143

@shayycisco as for your suggestion

Is it fair to say that these logs belong to debug? same as the unmarshall of the json stage

I'd argue that a structured log format like json or logfmt should not create a constant flow of parsing errors, what solely beats the purpose. So I much rather have the warning give me more info on what string and from which file could not be parsed for me to fix either the Promtail pipeline or my log format.

And if you really want to apply json or logfmt parsing on a "best effort" approach, there should be a switch for these pipeline stages to "ignore parsing errors" or to mark fields as "optional".

[dorkamotorka]

Any progress on this?

[Tristan971]

So I much rather have the warning give me more info on what string and from which file could not be parsed for me to fix either the Promtail pipeline or my log format.

This is the real answer in this case... The actual log line at fault being printed would really help a lot...

[galdor]

Same problem here, multiple failed to decode logfmt error messages which are completely useless since they do not contain the message that could not be parsed.

[frittentheke]

The original PR adding logfmt support was https://github.com/grafana/loki/pull/4346

@cyriltovena @jeschkies @kavirajk could you kindly take a look at this issue and my comment https://github.com/grafana/loki/issues/6066#issuecomment-1641771387? I gladly push a PR, but would love you opinions if you rather want:

1. info about which pipeline and message caused this error added to the log
2. ignore these errors (and only log this in debug level)
3. an option to ignore parsing errors within a pipeline and apply logfmt (or json for that matter) on a best-effort basis (maybe in conjunction with 1)

[rarrr]

Any update on this? I'd personally like the message added to the log and also ignore parsing errors.

[Kherm13]

I think logging to the debug level would be a huge improvement along with actually including the line (in only the debug level).