Graylog doesn't accept a newline as GELF short_message

hectorhuertas commented 6 years ago

Graylog is not accepting a newline "\n" as a valid GELF short_message, throwing an error message that is not very helpful.

Expected Behavior

I would prefer graylog to accept the newline, or to throw a warning instead of an error.

Many programs (I had the issue with mongodb) log empty lines and it's hard for general log collectors to parse and filter all the possible whitespace and non-useful strings. I understand the error if the field is not set, but maybe empty strings, whitespace and the like could be a warning or info instead of an error if you don't want to accept them.

What I would really like to see is a more descriptive error message and maybe better documentation on what is not allowed in the short_message field.

Current Behavior

Trying to ingest a gelf json with "short_message":"\n" throws an error

The only clue for debugging is the message has empty mandatory "short_message" field:

2018-06-12 14:40:45,775 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Unable to decode raw message RawMessage{id=972f15f0-6e4e-11e8-9b74-a28a23afbb85, journalOffset=17468033, codec=gelf, payloadSize=107, timestamp=2018-06-12T14:40:45.775Z, remoteAddress=/127.0.0.1:45498} on input <5b1f239f2e4b68000198bd9c>.
2018-06-12 14:40:45,775 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=972f15f0-6e4e-11e8-9b74-a28a23afbb85, journalOffset=17468033, codec=gelf, payloadSize=107, timestamp=2018-06-12T14:40:45.775Z, remoteAddress=/127.0.0.1:45498}
java.lang.IllegalArgumentException: GELF message <972f15f0-6e4e-11e8-9b74-a28a23afbb85> (received from <127.0.0.1:45498>) has empty mandatory "short_message" field.
        at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:252) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:134) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

I was collecting messages from a very busy fluentd instance, and it has been hard to find what kind of messages were throwing the errors, since I was looking blindly for empty strings.

Possible Solution

I have 3 suggestions:

Accept "\n" as valid short_message or lower its log level.
Print the full message in the error stack trace, at least in DEBUG or TRACE log levels. I think this will be most useful. It's very hard to know what messages are the problematic ones when you have a big volume of them
Could we have a more detailed description of valid short_messages? This is what the documentation says now (http://docs.graylog.org/en/2.4/pages/gelf.html#gelf-payload-specification):
```
short_message string (UTF-8)
a short descriptive message; MUST be set by client library.
```
Context

I am collecting logs from a kubernetes cluster using fluentd daemonset. I am sending the logs to graylog using https://github.com/bodhi-space/fluent-plugin-gelf-hs, which checks for the short_message being not empty, but is not parsing whitespace. I think now the plugin is doing right, since the docs doesn't specify what to check for.

Your Environment

Graylog Version: 2.4.5
Elasticsearch Version: 5.6.2
MongoDB Version: 3.4.6
Operating System:
Browser version:

Thank you very much!

juliohm1978 commented 4 years ago

I'm running into the exact same issue here. Fluentd sending logs to Graylog in a Kubernetes cluster. We have a large number of Java applications that spew empty lines in their logs, just "\n" all the time.

Rignt now, I'm having a hard time configuring fluentd to ignore these.

juliohm1978 commented 4 years ago

For anyone still facing this, I managed to configure fluentd to ignore blank lines.

        <filter **>
          @type grep
          <exclude>
            key log
            pattern ^\n$
          </exclude>
        </filter>

NOTES:

This filter must be placed before any <match> tags.

Also, keep in mind that dockerd creates logs in the following format:

{"log":"\n","stream":"stderr","time":"2020-03-25T18:08:47.256011014Z"}

So, if you are capturing logs from a different platform, your filter may not match the same log key from that input.

Graylog2 / graylog2-server