ERROR [DecodingProcessor] Unable to decode raw message RawMessage GL 3.3

commandline-be commented 4 years ago

With input GELF TCP receiving from NXLOG CE sending from Microsoft Windows 10 Pro there are no messages received visible and repeat errors occur on DecodingProcesor and ProcessBufferProcessor

Expected Behavior

No ERRORS

Current Behavior

2020-06-17T23:27:02.298+02:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=<obfuscated>-b0e1-11ea-b2e1-7e053dcd3d04, journalOffset=404488, codec=CEF, payloadSize=502, timestamp=2020-06-17T21:27:02.296Z, remoteAddress=/192.168.<obfuscated>:37154} on input <5eea489a2c824e0cff01f2bb>. 2020-06-17T23:27:02.298+02:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=<obfuscated>-b0e1-11ea-b2e1-7e053dcd3d04, journalOffset=404488, codec=CEF, payloadSize=502, timestamp=2020-06-17T21:27:02.296Z, remoteAddress=/192.168.<obfuscated>:37154} java.lang.NullPointerException: null at org.graylog.plugins.cef.parser.MappedMessage.(MappedMessage.java:37) ~[graylog.jar:?] at org.graylog.plugins.cef.codec.CEFCodec.decodeCEF(CEFCodec.java:128) ~[graylog.jar:?] at org.graylog.plugins.cef.codec.CEFCodec.decode(CEFCodec.java:117) ~[graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?] at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?] at java.lang.Thread.run(Thread.java:834) [?:?]

Possible Solution

nothing relevant was found, yet may be timestamp related

Steps to Reproduce (for bugs)

enable GELF TCP input
enabled NXLOG CE gelf output
show received messages
tail /var/log/graylog-server/server.log

Context

https://community.graylog.org/t/error-decodingprocessor-unable-to-decode-raw-message-rawmessage-gelf-with-nxlog/15983/4

Your Environment

Graylog Version: 3.3.1-5
Elasticsearch Version: 6.8.10 (oss)
MongoDB Version: 4.2.8
Operating System: Debian Buster
Browser version: Brave (latest)

Input Metrics

org.graylog.plugins.cef.input.CEFTCPInput.5eea489a2c824e0cff01f2bb.rawSize Meter Total: 47,760,610 events Mean: 32,419.95 events/second 1 minute avg: 1.17 events/second 5 minute avg: 8,629.5 events/second 15 minute avg: 20,071.24 events/second

org.graylog.plugins.cef.codec.CEFCodec.5eea489a2c824e0cff01f2bb.failures Meter Total: 361,432 events Mean: 21.09 events/second 1 minute avg: 0.26 events/second 5 minute avg: 14.25 events/second 15 minute avg: 51.01 events/second

org.graylog2.indexer.messages.Messages.invalid-timestamps Meter Total: 0 events Mean: 0 events/second 1 minute avg: 0 events/second 5 minute avg: 0 events/second 15 minute avg: 0 events/second

bernd commented 4 years ago

@commandline-be According to the error message, you are sending your nxlog data to a CEF input. (org.graylog.plugins.cef.codec.CEFCodec) Please check if you configured nxlog to send data to the correct port.

commandline-be commented 4 years ago

Thanks. I wonder a about exactly that.

I used a GELF input and send using GELF to a custom port.

commandline-be commented 4 years ago

Looking back i am not certain what is going on. I deleted and recreated the GELF TCP input. Now it shows very different settings. Not at all understanding why.

bernd commented 4 years ago

@commandline-be Hmm, strange. Does it work now or do you still think there is a bug somewhere?

Thanks!

no-response[bot] commented 4 years ago

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

1k-off commented 2 years ago

Seems, that I managed to reproduce this.

Inputs:

typical docker-compose file:

version: '3'
services:
mongo:
  image: mongo:4.2
  networks:
    - graylog
  restart: always
elasticsearch:
  image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
  restart: always
  environment:
    - http.host=0.0.0.0
    - transport.host=localhost
    - network.host=0.0.0.0
    - "ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true -Xms512m -Xmx512m"
  ulimits:
    memlock:
      soft: -1
      hard: -1
  deploy:
    resources:
      limits:
        memory: 1g
  networks:
    - graylog
graylog:
  image: graylog/graylog:4.2
  restart: always
  environment:
    - GRAYLOG_PASSWORD_SECRET=masked
    - GRAYLOG_ROOT_PASSWORD_SHA2=masked
    - GRAYLOG_HTTP_EXTERNAL_URI=http://192.168.1.102:9000/
  entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
  networks:
    - graylog
  restart: always
  depends_on:
    - mongo
    - elasticsearch
  ports:
    - 9000:9000
    - 1514:1514
    - 1514:1514/udp
    - 12201:12201
    - 12201:12201/udp
networks:
graylog:
  driver: bridge

docker service writing logs to graylog

docker run -itd --name foo -v $(pwd)/data:/app/data -p 8080:8080 --log-driver=gelf --log-opt gelf-address=udp://192.168.1.102:12201 --restart always local/bar:latest

Steps to reproduce:

Launch graylog with default docker-compose setup
Login to graylog admin
Go to System -> Inputs
Create CEF UDP input with GELF port (https://i.imgur.com/qbTnDrc.png)
Start service that must send logs to graylog
Leave it for 3-5 minutes
Delete this input
Create GELF UDP input with the same port (12201) (https://i.imgur.com/hdWgbbl.png)
Restart (or not) docker service.

Check graylog logs

2022-01-23 21:57:09,498 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=cc25bf80-7c95-11ec-a344-0242ac130004, messageQueueId=98521, codec=CEF, payloadSize=454, timestamp=2022-01-23T21:45:35.864Z, remoteAddress=/192.168.3.109:32777}
java.lang.NullPointerException: null
    at org.graylog.plugins.cef.parser.MappedMessage.<init>(MappedMessage.java:37) ~[graylog.jar:?]
    at org.graylog.plugins.cef.codec.CEFCodec.decodeCEF(CEFCodec.java:128) ~[graylog.jar:?]
    at org.graylog.plugins.cef.codec.CEFCodec.decode(CEFCodec.java:117) ~[graylog.jar:?]
    at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) ~[graylog.jar:?]
    at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:95) [graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
    at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]

Graylog2 / graylog2-server