search

opensearch-project / logstash-output-opensearch

A Logstash plugin that sends event data to a OpenSearch clusters and stores as an index.
https://opensearch.org/docs/latest/clients/logstash/index/
Apache License 2.0
107 stars 80 forks source link

How can I keep the connections in the connection pool always alive? #257

Open donggyu04 opened 2 months ago

donggyu04 commented 2 months ago

If there is no request for a certain amount of idle time for OpenSearch, an error occurs when a request is sent to Logstash afterward.

Attempted to send a bulk request but there are no living connections in the pool (perhaps OpenSearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>16}

The OpenSearch server has no problem, and the requests are processed successfully as the connection is re-established during the retry process.

It seems that the stale connections in the connection pool are not being refreshed properly. In this cases, is there a way to manage the connections in the connection pool to ensure they are always kept alive?

There is a validate_after_inactivity setting, but it doesn't seem to work as I want it to. Would enabling sniffing help in this case?

Thanks in advance.

dblock commented 2 months ago

This sounds like a bug, will likely require more digging :(

donggyu04 commented 2 months ago

@dblock

Hi, Thanks for your reply. It is not easy to reproduce this phenomenon.

in my case, this occurred when logstash was idle for a long time without any calls after connecting to OpenSearch. You may be able to reproduce it by calling a few hours or a day later (This may vary depending on the environment)