search

fluent / fluentd-kubernetes-daemonset

Fluentd daemonset for Kubernetes and it Docker image
Apache License 2.0
1.27k stars 980 forks source link

GELF messages to Graylog server doesn't contain mandatory field - short_message #243

Open eladtamary opened 5 years ago

eladtamary commented 5 years ago

Hi, We are using the daemonset to send logs to centralized Graylog server using the following image: fluent/fluentd-kubernetes-daemonset:v1.2-debian-graylog.

The logs are sent to Graylog and we are able to filter them properly. However, we saw repetitive errors in Graylog server logs about missing mandatory field - short_message. We understood that this is a mandatory field in GELF protocol that must be sent from any client.

How do I make the daemonset send this field in the GELF message.

Thanks, Elad Tamary

shinebayar-g commented 5 years ago

Hi, I've been using fluent/fluentd-kubernetes-daemonset:v1.3-debian-graylog image and I believe I didn't get this issue. I'd suggest try updated image.

de1m commented 5 years ago

I've tested with both images fluent/fluentd-kubernetes-daemonset:v1.4-debian-graylog-1 and fluent/fluentd-kubernetes-daemonset:v1.4.2-debian-graylog-1.1. But I get the same error. Ps. I use the graylog 3.1 from this docker file

myspotontheweb commented 5 years ago

This is issue is being continually closed as a docker error. I'm wondering if it's actually an issue with the handling of the GELF message as reported here (logging an empty line):

https://github.com/Graylog2/graylog2-server/issues/4842

shinebayar-g commented 5 years ago

Update: I just noticed I'm getting this error on graylog server console as well. So is there any side effects besides this error messages?

2019-08-20 13:22:06,229 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Unable to decode raw message RawMessage{id=8164dd48-c34d-11e9-b7e7-0242ac11000e, journalOffset=720106792, codec=gelf, payloadSize=554, timestamp=2019-08-20T13:22:06.228Z, remoteAddress=/XX.XX.XX.XX:36090} on input <5d3ec6aa6b2f07000fb685da>.
2019-08-20 13:22:06,229 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=8164dd48-c34d-11e9-b7e7-0242ac11000e, journalOffset=720106792, codec=gelf, payloadSize=554, timestamp=2019-08-20T13:22:06.228Z, remoteAddress=/XX.XX.XX.XX:36090}
java.lang.IllegalArgumentException: GELF message <8164dd48-c34d-11e9-b7e7-0242ac11000e> (received from <XX.XX.XX.XX:36090>) has empty mandatory "short_message" field.
    at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:252) ~[graylog.jar:?]
    at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:134) ~[graylog.jar:?]
    at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
    at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:86) [graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:45) [graylog.jar:?]
    at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
chuegel commented 5 years ago

I have the same issue. Rancher -> fluentd -> Graylog GELF TCP input

Rancher 2.2.4 Graylog 3.1.

chuegel commented 5 years ago

Exporting logs from Rancher to Graylog via fluentd is not supported yet. See: https://github.com/rancher/rancher/issues/23052

repeatedly commented 5 years ago

Exporting logs from Rancher to Graylog via fluentd is not supported yet. See:

This issue says "Rancher can't export data to Graylog directly". fluentd seems not related.

robermar23 commented 4 years ago

I am seeing the same error. using image fluent/fluentd-kubernetes-daemonset:v1.7.4-debian-graylog-2.2

fluentd daemonset running on every node, using gelf, sending to graylog 3.2.2.

ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=8d22033b-6535-11ea-b2aa-0a580a8102bb, journalOffset=-9223372036854775808, codec=gelf, payloadSize=1168, timestamp=2020-03-13T14:18:46.371Z, remoteAddress=****}
  | java.lang.IllegalArgumentException: GELF message <8d22033b-6535-11ea-b2aa-0a580a8102bb> (received from ****) has empty mandatory "short_message" field.
HaveFun83 commented 4 years ago

same here

ismailyenigul commented 4 years ago

and I have the same problem on Graylog 3.3 with ES 6.8 using fluentd-daemonset-graylog-rbac.yaml

aylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
2020-05-23T19:40:31.771Z ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=8e0e95a2-9d26-11ea-b135-1a3420eca63d, journalOffset=119788096, codec=gelf, payloadSize=561, timestamp=2020-05-23T18:52:30.586Z, remoteAddress=/10.135.210.216:35821} on input <5ec5b96fabdddd32c54deee6>.
2020-05-23T19:40:31.771Z ERROR [DecodingProcessor] Error processing message RawMessage{id=8e0e95a2-9d26-11ea-b135-1a3420eca63d, journalOffset=119788096, codec=gelf, payloadSize=561, timestamp=2020-05-23T18:52:30.586Z, remoteAddress=/10.135.210.216:35821}
java.lang.IllegalArgumentException: GELF message <8e0e95a2-9d26-11ea-b135-1a3420eca63d> (received from <10.135.210.216:35821>) has empty mandatory "short_message" field.
        at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:258) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
@

If I restart graylog server, log flow starts again.

shizacat commented 4 years ago

1.11 I have the same problem on Graylog 3.3

graylog_1  | 2020-08-04 11:53:34,187 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=fa72b590-d62f-11ea-9db4-3ace9f95a535, journalOffset=1974010089, codec=gelf, payloadSize=545, timestamp=2020-08-04T08:53:34.185Z, remoteAddress=/172.19.103.133:38921}
graylog_1  | java.lang.IllegalArgumentException: GELF message <fa72b590-d62f-11ea-9db4-3ace9f95a535> (received from <172.19.103.133:38921>) has empty mandatory "short_message" field.
graylog_1  |    at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:258) ~[graylog.jar:?]
graylog_1  |    at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140) ~[graylog.jar:?]
graylog_1  |    at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
graylog_1  |    at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
graylog_1  |    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
graylog_1  |    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
graylog_1  |    at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
graylog_1  |    at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
graylog_1  |    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
ediezh commented 3 years ago

I have the same problem despite I added the below filter in the fluentd config.

    <filter **>
      @type grep
      <exclude>
        key log
        pattern ^\n$
      </exclude>
    </filter>
nix-power commented 3 years ago

Hi, We are using the daemonset to send logs to centralized Graylog server using the following image: fluent/fluentd-kubernetes-daemonset:v1.2-debian-graylog.

The logs are sent to Graylog and we are able to filter them properly. However, we saw repetitive errors in Graylog server logs about missing mandatory field - short_message. We understood that this is a mandatory field in GELF protocol that must be sent from any client.

How do I make the daemonset send this field in the GELF message.

Thanks, Elad Tamary

I am working on a workaround to resolve it.

danielfm commented 2 years ago

Did anyone find any workaround for eliminating these errors?

zolech commented 2 years ago

@nix-power Did you find any workaround ?