Open mahnunchik opened 5 years ago
Possible related to https://github.com/Graylog2/graylog2-server/issues/6488
I am getting the same after migration to Graylog 3.1.
I can confirm this Bug. But I have other circumstances. Using nxlog-ce to send GELF_TCP. I also wrote the GELF_TCP-Message to a file an got this:
{"version":"1.1",
"_EventReceivedTime":"2020-08-24 11:04:02",
"_SourceModuleName":"redmine_production",
"_SourceModuleType":"im_file",
"_SourceName":"Redmine Production",
"host":"VEP01",
"short_message":"Started GET \"**********" for 127.0.0.1 at 2020-08-24 11:04:01 +0200\r\nProcessing by TimeLoggersController#render_menu as */*\r\n Parameters: {\"_\"=>\"1598252400620\"}\r\n Current user: **** (id=**)\r\n Rendered ******************** (11.3ms)\r\nCompleted 200 OK in 20ms (Views: 9.9ms | ActiveRecord: 5.7ms)\r",
"timestamp":1598259842,
"level":6}
As you can see the short_message is there. But graylog says:
2020-08-24T11:04:02.121+02:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=**********, journalOffset=260784, codec=gelf, payloadSize=233, timestamp=2020-08-24T09:04:02.120Z, remoteAddress=/192.168.1.***:61723} on input <5f40331381d87961ab292a56>.
2020-08-24T11:04:02.121+02:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=c0fcfc81-e5e8-11ea-aa55-00505681aa85, journalOffset=260784, codec=gelf, payloadSize=233, timestamp=2020-08-24T09:04:02.120Z, remoteAddress=/192.168.1.***:61723}
java.lang.IllegalArgumentException: GELF message <*****************> (received from <192.168.1.***:61723>) has empty mandatory "short_message" field.
at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:258) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
Also noteable might be, the message is prcessed correctly by graylog.
Maybe this helps to solve this issue.
My graylog logs are filled with these message. It's about 10K lines each minute.
Maybe recreate the input is a workaround.
I have the same problem with my Graylog 4.0.11+e4e88a4, codename Noir. Any update?
Here are the logs :
2022-01-14 09:48:04,082 ERROR [DecodingProcessor] - Unable to decode raw message RawMessage{id=119053b1-751f-11ec-96f4-7af235b1e3ab, journalOffset=198692353, codec=gelf, payloadSize=514, timestamp=2022-01-14T09:48:04.075Z, remoteAddress=/10.40.0.0:19396} on input <607e93166eddf7582244c00f>. - {} 2022-01-14 09:48:04,082 ERROR [DecodingProcessor] - Error processing message RawMessage{id=119053b1-751f-11ec-96f4-7af235b1e3ab, journalOffset=198692353, codec=gelf, payloadSize=514, timestamp=2022-01-14T09:48:04.075Z, remoteAddress=/10.40.0.0:19396} - {} java.lang.IllegalArgumentException: GELF message <119053b1-751f-11ec-96f4-7af235b1e3ab> (received from <10.40.0.0:19396>) has empty mandatory "short_message" field. at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:258) ~[graylog.jar:?] at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140) ~[graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:149) ~[graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:90) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?] at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
The same on Graylog v4.2.6. Any updates on the subj?
same here with v4.3.3
Same here with v4.3.8
I'm also receiving this on GrayLog v6.0
Current Behavior
I've a lot of errors in log:
Possible Solution
It is known problem of docker https://github.com/docker/for-linux/issues/354
Your Environment