search

opensearch-project / logstash-output-opensearch

A Logstash plugin that sends event data to a OpenSearch clusters and stores as an index.
https://opensearch.org/docs/latest/clients/logstash/index/
Apache License 2.0
104 stars 80 forks source link

[QUESTION] Clarification on Data Processing Order During Outages #241

Open nw-engineer opened 6 months ago

nw-engineer commented 6 months ago

Hello,

I hope this message finds you well. I am currently utilizing Logstash 8.4.3 with the OpenSearch output plugin for my data pipeline and have some questions regarding its behavior during network outages or service disruptions.

I understand that in the event of a connectivity issue to OpenSearch, the plugin is designed to retry indefinitely. While I recognize the importance of ensuring data delivery, I am concerned about the potential implications of this behavior during prolonged outages. Specifically, I am interested in understanding how Logstash prioritizes data processing and retry attempts under such circumstances.

Could you please clarify if, during a prolonged outage where Logstash cannot connect to OpenSearch, Logstash prioritizes retrying the buffered or queued data over processing new incoming data? In other words, does Logstash attempt to clear the backlog of retries before it starts processing newly received data once the connection is re-established?

This information is crucial for planning our data pipeline resilience and understanding how Logstash would handle scenarios where the target index might become unavailable (e.g., moved to UltraWarm in AWS Elasticsearch Service) and further complicates the retry logic.

Thank you for your time and assistance in addressing this query. Your insights will be incredibly valuable for our ongoing and future implementations.

Best regards

dblock commented 6 months ago

This is really a question for the authors of logstash, however I think it would be useful to document the behavior in this plugin, so let's leave this open. @nw-engineer if you find answers to these will you please help with that?

nw-engineer commented 6 months ago

@dblock Thank you very much for your prompt and insightful response. I agree that documenting the behavior in the plugin could greatly benefit the community, and I'm more than willing to contribute to this effort.

Thank you again for your guidance and for considering the value of documenting this behavior. I look forward to contributing to the community and helping improve the resources available to all users.