search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.4k stars 1.07k forks source link

Neither GELF UDP nor SYSLOG UDP work with NGINX JSON Logs #14376

Closed hackdefendr closed 1 year ago

hackdefendr commented 1 year ago

Expected Behavior

In the last stable v4 releases everything worked properly. In v5.0.2, I cannot get properly formatted JSON Nginx logs to ingest into Graylog v5.0.2. Since I am using an Nginx template to format the logs, I assumed I should use GELF UDP for the input. When I do, none of the messages get through because they are dropped due to these errors:

2023-01-08T18:16:17.587-06:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=d603b030-8fb2-11ed-b935-d604744095b0, messageQueueId=482, codec=gelf, payloadSize=371, timestamp=2023-01-09T00:16:17.587Z, remoteAddress=/192.168.10.87:53806} on input <63bb5b3a5453ca334d9b1408>.
2023-01-08T18:16:17.587-06:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=d603b030-8fb2-11ed-b935-d604744095b0, messageQueueId=482, codec=gelf, payloadSize=371, timestamp=2023-01-09T00:16:17.587Z, remoteAddress=/192.168.10.87:53806}
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (String)"<190>Jan  9 00:16:17 unpopular nginx: {"source": "nginx", "time": 1673223377.547, "resp_body_size": 5, "host": "unpopular.cloud", "address": "195.201.108.96", "request_length": 5012, "method": "POST", "uri": "/inbox", "status": 202,  "user_agent": "http.rb/5.1.1 (Mastodon/4.0.2+glitch; +https://infosec.exchange/)", "resp_time": 0.011, "upstream_addr": "127.0.0.1:3000"}"; line: 1, column: 2]
    at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391) ~[graylog.jar:?]
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:735) ~[graylog.jar:?]
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:659) ~[graylog.jar:?]
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:2005) ~[graylog.jar:?]
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:802) ~[graylog.jar:?]
    at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4703) ~[graylog.jar:?]
    at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3076) ~[graylog.jar:?]
    at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
    at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:156) ~[graylog.jar:?]
    at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:95) [graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
    at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
    at java.lang.Thread.run(Unknown Source) [?:?]

Here is my Nginx template:

log_format json_logs escape=json
  '{'
    '"time_local":"$time_local",'
    '"remote_addr":"$remote_addr",'
    '"remote_user":"$remote_user",'
    '"request":"$request",'
    '"status": "$status",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"request_time":"$request_time",'
    '"http_referrer":"$http_referer",'
    '"http_user_agent":"$http_user_agent"'
  '}';
access_log /var/log/access.log json_logs;

Current Behavior

Switching the input to SYSLOG UDP simply does not work at all. I see no errors in the logs nor do I see messages hitting the input.

Possible Solution

I do not really know what changed, but it definitely worked in the last v4 release.

Steps to Reproduce (for bugs)

  1. Configure Nginx logging format using any template you wish.
  2. Configure GELF UDP input in Graylog and start it.
  3. Configure Nginx access log to send to above GELF UDP input.
  4. Tail and Follow /var/log/graylog/server.log
  5. Generate some traffic and watch the errors pour in.

Context

We used to have some pretty sweet dashboards in v4 tracking multiple items and events from multiple Nginx servers. Now, everything is empty except the graylog server log. I already mentioned this, but switching to SYSLOG UDP is not the answer because that input doesn't work either, and there is no log showing why.

Your Environment

rkmbaxed commented 1 year ago

Your log <190>Jan 9 00:16:17 unpopular nginx: {"source": "nginx", "time": 1673223377.547, "resp_body_size": 5, "host": "unpopular.cloud", "address": "195.201.108.96", "request_length": 5012, "method": "POST", "uri": "/inbox", "status": 202, "user_agent": "http.rb/5.1.1 (Mastodon/4.0.2+glitch; +https://infosec.exchange/)", "resp_time": 0.011, "upstream_addr": "127.0.0.1:3000"} has in my opinion no valid GELF format. https://go2docs.graylog.org/5-0/getting_in_log_data/gelf.html

Syslog: https://go2docs.graylog.org/5-0/getting_in_log_data/ingest_syslog.html Did you try to ingest with a RAW/Plaintext UDP?

Also you have no entrys in the stream "Processing and Indexing Failures"?

hackdefendr commented 1 year ago

Right that is the full syslog. This is the message body:

 {"source": "nginx", "time": 1673223377.547, "resp_body_size": 5, "host": "unpopular.cloud", "address": "195.201.108.96", "request_length": 5012, "method": "POST", "uri": "/inbox", "status": 202, "user_agent": "http.rb/5.1.1 (Mastodon/4.0.2+glitch; +https://infosec.exchange/)", "resp_time": 0.011, "upstream_addr": "127.0.0.1:3000"}

That is JSON formatted from Nginx and it used to ingest just like that in v4.

hackdefendr commented 1 year ago

OK an update.

I finally got Syslog UDP working for both Access and Error Nginx logs. Access logs are the only ones where Nginx supports converting to JSON. I still feel that GELF should work with the Access logs when they are properly formatted JSON logs, which they are, but are embedded within a syslog messages.

J

patrickmann commented 1 year ago

@hackdefendr Glad to hear you were able to solve the issue. It's an interesting idea to chain codecs like that; but not on the road map I'm afraid.

On a side note: I still don't quite understand how this nested message could have been ingested in v4. I assume you were not sending it inside a SYSLOG? Also, the sample you provided isn't valid GELF, since it lacks the mandatory version and short_message fields.

I am closing this issue, since there isn't anything actionable for us.

fomcl commented 1 year ago

OK an update.

I finally got Syslog UDP working for both Access and Error Nginx logs. Access logs are the only ones where Nginx supports converting to JSON. I still feel that GELF should work with the Access logs when they are properly formatted JSON logs, which they are, but are embedded within a syslog messages.

J

Can you explain how you solved this? I've been struggling with this in the exact same way. One Datalust blog mentioned that GELF is the recommended way, but I saw the same gelf-inside-syslog as you did. I also tried syslog RFC 5424 and RFC 3164. The former using a custom log format, while the latter is supposed to be the nginx default. In both cases, the nginx log was one chunk. I could not query the fields.

Note: I use Datalust Seq, not Graylog. Nonetheless I hope your solution will be of use to me.

saper44rus commented 8 months ago

@hackdefendr Glad to hear you were able to solve the issue. It's an interesting idea to chain codecs like that; but not on the road map I'm afraid.

On a side note: I still don't quite understand how this nested message could have been ingested in v4. I assume you were not sending it inside a SYSLOG? Also, the sample you provided isn't valid GELF, since it lacks the mandatory version and short_message fields.

I am closing this issue, since there isn't anything actionable for us.

I faced the same problem, how did you solve it? Is JSON parsed or is it just one line?