opensearch-project / logstash-output-opensearch

A Logstash plugin that sends event data to a OpenSearch clusters and stores as an index.
https://opensearch.org/docs/latest/clients/logstash/index/
Apache License 2.0
106 stars 80 forks source link

[BUG] HttpClient's client_settings cannot be configured #134

Open piequi opened 2 years ago

piequi commented 2 years ago

Describe the bug client_settings aren't configurable from output plugin configuration.

To Reproduce Steps to reproduce the behavior:

  1. Run Docker images locally like described in https://opensearch.org/docs/latest/clients/logstash/index/#docker
  2. Add client_settings key as described in https://github.com/opensearch-project/logstash-output-opensearch/blob/aeee1cfbf5135fc23a6f9a2ef57ab24b7d55ca04/lib/logstash/outputs/opensearch/http_client.rb#L38 and https://github.com/opensearch-project/logstash-output-opensearch/blob/aeee1cfbf5135fc23a6f9a2ef57ab24b7d55ca04/lib/logstash/outputs/opensearch/http_client.rb#L299
    opensearch {
    hosts => ["https://opensearch:9200"]
    index => "opensearch-logstash-docker-%{+YYYY.MM.dd}"
    user => "admin"
    password => "admin"
    ssl => true
    ssl_certificate_verification => false
    client_settings => {
    check_connection_timeout => 30
    }
    }
  3. Logstash will fail starting stating that [ERROR][logstash.outputs.opensearch] Unknown setting 'client_settings' for opensearch

Expected behavior Logstash should start without any issue with the provided settings applied to HttpClient

Plugins logstash-output-opensearch (1.2.0)

Host/Environment (please complete the following information):

Additional context We are in a situation where network latency is very high between Logstash instances and Opensearch cluster. They actually run on distant AWS regions. timeout plugin setting is available but doesn't seem to update the right parameter as we see

[DEBUG][org.apache.http.conn.ssl.SSLConnectionSocketFactory][main] Connecting socket to vpc-elk-xxx.es.amazonaws.com/10.x.x.x:443 with timeout 10000

and we cannot change those 10 seconds whatever the value we provide in timeout

piequi commented 2 years ago

Actually check_connection_timeout wouldn't help; the parameter we need to update instead would be connect_timeout : https://github.com/cheald/manticore/blob/8d5e4768bf898fc2d87cf8ee7bbb585bce0693ec/lib/manticore/client.rb#L146

piequi commented 2 years ago

https://github.com/opensearch-project/logstash-output-opensearch/blob/aeee1cfbf5135fc23a6f9a2ef57ab24b7d55ca04/lib/logstash/outputs/opensearch/http_client.rb#L292 Could be reconsidered and so that the values are taken from the provided client_settings instead. This would allow a more precise and transparent configuration.