search

logstash-plugins / logstash-output-elasticsearch

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
Apache License 2.0
219 stars 305 forks source link

Add Elastic Api Version header #1147

Closed kaisecheng closed 1 year ago

kaisecheng commented 1 year ago

This commit adds "Elastic-Api-Version" : "2023-10-31" to request header when the Elasticsearch is serverless Rename "flavour" to "flavor"

kaisecheng commented 1 year ago

The updates handle custom_headers with an invalid version. The healthcheck! do two requests in the register phase. 1. health check request 2. fetch the info from main endpoint /. The second request is needed to get the failure reason as the first gives empty body. The commits add response_body to log if data exists.

I have two thoughts on handling the failures

  1. ~We can add a test request in the register phase to make sure elastic api version in hand is useable. The plugin hardcode "2023-10-31" and I don't expect server returns failure for this version. A test request can make sure everything is set and can stop the pipeline fast when thing goes wrong.~ Currently, changing the version to "2024" in the source code won't shutdown the pipeline as it is after register. Edit: I will make the pipeline termination in another PR to unify the error behavior. When ConfigurationError is raised in finish_register(), the pipeline should stop.
  2. bulk request could get 400 invalid api version and retry indefinitely. Other 4xx and 5xx use the same retry mechanism. I don't think we want to shutdown the pipeline in this case.
kaisecheng commented 1 year ago

summary: we decided to add a get "/" request in healthcheck! in register phase to make sure the first health check would fail when the header is invalid