agido-malter / logback-elasticsearch-appender

Logback Elasticsearch Appender
Other
23 stars 11 forks source link

Have you fixed "The bulk request must be terminated by a newline"? #10

Closed gavenkoa closed 1 year ago

gavenkoa commented 1 year ago

https://github.com/internetitem/logback-elasticsearch-appender/issues/63 https://github.com/internetitem/logback-elasticsearch-appender/issues/86

We are using this logger for years and sometimes internal state machine fails, the logger attempts to deliver invalid data infinitely:

WARN in com.internetitem.logback.elasticsearch.util.ErrorReporter@5b03fb49 - Failed to send events to Elasticsearch: Got response code [400] from server with data {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"},"status":400}

I think there are cases with non-null value as during long run it is still entering infinite broken data delivery loop. Like fix should targets the internal state machine and allows recovery path based on ES response. It is fine to lose few messages, it is more dangerous to be stuck infinitely.

Original project is dead, I thought about getting rid of Logback ES appender in favor to Fluentbit log shipper.

gavenkoa commented 1 year ago

The only solution to recover is server rebooting ((

agido-jrieks commented 1 year ago

@gavenkoa thank you for the report, I'll fix the problem. please note that log.info(null) could be also be seen as an incorrect API usage: if you have an arbitrary string, you should probably prefer to use log.info("{}", whatever) instead - if the string would contain "{}", it would be interpreted as an argument...

agido-malter commented 1 year ago

Fixed in 3.0.5