jortkoopmans
commented
2 years ago Hi @Zava2012 , just stumbled across your ticket while looking for multiline processing myself (somewhat similar situation). Not an expert, but my guess is that you're observing your regex not matching the multiline logs.
To clarify, expecting these steps of processing;
- Convert your docker logs, which are json, extracting the timestamp and stream labels. See also locally in your promtail containers under /var/log/pods/*.
- Multiline stage does as you describe, but all messages are passed on to further stages (whether touched by multiline or not).
- in regex stage, loglines which are single-line match this statement (as you have ^ and $), while multiline does not.
- Only single-line loglines then have the extracted map which you can convert into labels.
You could try to modify your regex to also work on multiline logs with a robust extracted map, but it might prove tricky.
For reference, I'm trying something similar with a regex, but then using the &ZeroWidthSpace character, using a match block and a custom extracted label to match on.
- docker: {}
- multiline:
# Use this special stage to combine specifically tagged lines into one multiline message
firstline: '^​'
max_wait_time: 3s
max_lines: 192
- regex:
# This regex adds the 'multiline' content to the extracted map
expression: '^(?s)(?P<multiline>​)'
- labels:
# Now label messages that have a 'multiline' in their extracted map
multiline:
- match:
# Only filter messages that are multiline labelled
selector: '{multiline="​"}'
stages:
- replace:
# Clean up zerospacewidth
expression: '^(?s)(?P<zerowidthspace>​)'
replace: ''
- replace:
# Remove empty lines
expression: '(?m)(?P<emptyline>^\s*\n)'
replace: ''Hope it helps! 🤞
Zava2012
jeschkies
stale[bot]
cstyan
Describe the bug multiline block is catching all of the logs including the one-line log. I'll provide an example in the next section to clarify all of it.
To Reproduce There are two blocks of logs structure. They were taken from the real application. It means all timestamps and strings exactly repeat logs from the application. The first block of logs (this is one-line log):
The second block of logs (stacktrace for Python):
On the Promtail side I have this simple configuration to catch stacktraces of Python applications. You can find almost the same configuration in official documentation
And all INFO and ERROR logs are written to the "error" label.
Expected behavior As far as I understand, "firstline" field in the multiline block says if the next line of the log doesn't match a regex in "firstline" field then "multiline" block concatenate the next log line with a previous line of the log. The next lines of logs work in the same way and form an extracted map that can be caught by the next "regex" block without catching all of the logs via "regex" block.
How can I write only stacktraces to the label without one-line ERROR and INFO logs?
Environment: