Closed osuff closed 1 year ago
Thanks @osuff for reporting this!
How can we reproduce this error?
Per my limited understanding, this is just how Graylog stores messages in ES, I am not sure we are able to affect this part of Graylog. In our environment, these empty messages appear quite frequently when we do a tcpdump of communication between Graylog and ES.
I've tested disabling our single drop_message rule in the pipeline, which did not help. Other rules in pipelines are only enriching messages, or routing to streams. There are no errors or anomalies in the logfiles which I could see.
This is hard to track fully as the offending part of the json is lacking any unique identifier of the empty message.
Closing due to being outdated. Please reopen if this is still an issue for you!
When using bulk messages, Graylog includes empty messages. This behaviour results in ES responding with error code 400, which is not propagated to the server.log. Outcome of this parsing error is that EC does not store messages from the bulk update to the index, therefore loosing the data silently.
Expected Behavior
Graylog does not send incorrect jsons to ES Response 400 from ES results in an entry in the logfile.
Current Behavior
Graylog sends bulk updates containing "empty" messages per sample below, this is reported by ES Response 400 is not visibly propagated to application logs
Possible Hotfix
Set bulk update size to 1. Although Graylog sends empty messages to ES, no other messages are lost.
Example of communication
Using bulk update of 5 messages
Request:
Response: