logstash-plugins / logstash-output-elasticsearch

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
Apache License 2.0
219 stars 305 forks source link

Too large payload responses (413) are not being properly handled #1180

Open edmocosta opened 4 months ago

edmocosta commented 4 months ago

Logstash is crashing with OOM when documents are bigger than the Elasticsearch http.max_content_length. Those documents are filling up the memory and are never freed due to infinite retries.

it seems there’s a clear intention to handle 413 status here, which is not working properly due to the manticore's adapter logic. This adapter raises an error, making the http_client condition unreachable, leading to infinite retries and no options to DLQ those problematic events (dlq_custom_codes).


Relates to: https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/785