Graylog bulk update sends incorrect json messages to ES

[osuff]

When using bulk messages, Graylog includes empty messages. This behaviour results in ES responding with error code 400, which is not propagated to the server.log. Outcome of this parsing error is that EC does not store messages from the bulk update to the index, therefore loosing the data silently.

Expected Behavior

Graylog does not send incorrect jsons to ES Response 400 from ES results in an entry in the logfile.

Current Behavior

Graylog sends bulk updates containing "empty" messages per sample below, this is reported by ES Response 400 is not visibly propagated to application logs

Possible Hotfix

Set bulk update size to 1. Although Graylog sends empty messages to ES, no other messages are lost.

Example of communication

Using bulk update of 5 messages

Request:

POST /_bulk HTTP/1.1
Content-Length: 4289
Content-Type: application/json; charset=UTF-8
Host: 127.0.0.1:9200
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.4 (Java/1.8.0_172)
Accept-Encoding: gzip,deflate

{"index":{"_id":"dccfae80-aa94-11e8-9b4a-5254000f0f0f","_index":"mit_deflector","_type":"message"}}
{"index":{"_id":"dccf8773-aa94-11e8-9b4a-5254000f0f0f","_index":"mit_deflector","_type":"message"}}
{ REDACTED MESSAGE CONTENTS "facility":"filebeat","ts":"1535441695.391670"}
{"index":{"_id":"dccfae85-aa94-11e8-9b4a-5254000f0f0f","_index":"mit_deflector","_type":"message"}}
{ REDACTED MESSAGE CONTENTS "facility":"filebeat","ts":"1535441680.083583"}
{"index":{"_id":"dccfae82-aa94-11e8-9b4a-5254000f0f0f","_index":"mit_deflector","_type":"message"}}
{"index":{"_id":"dccfae81-aa94-11e8-9b4a-5254000f0f0f","_index":"mit_deflector","_type":"message"}}

Response:

HTTP/1.1 400 Bad Request
content-type: application/json; charset=UTF-8
content-encoding: gzip
content-length: 188

@EexZ(@"o=[4 ,fPFk|G=+4
$8qnhj:tX!0mOR4Qrao<3>b99

• Graylog Version: Graylog 2.4.6+ceaa7e4
• Elasticsearch Version: 5.6.11_1
• MongoDB Version: 2.6.12_6.el7
• Operating System: n/a
• Browser version: n/a

[dennisoelkers]

Thanks @osuff for reporting this!

How can we reproduce this error?

[osuff]

Per my limited understanding, this is just how Graylog stores messages in ES, I am not sure we are able to affect this part of Graylog. In our environment, these empty messages appear quite frequently when we do a tcpdump of communication between Graylog and ES.

I've tested disabling our single drop_message rule in the pipeline, which did not help. Other rules in pipelines are only enriching messages, or routing to streams. There are no errors or anomalies in the logfiles which I could see.

This is hard to track fully as the offending part of the json is lacking any unique identifier of the empty message.

[dennisoelkers]

Closing due to being outdated. Please reopen if this is still an issue for you!