search

opensearch-project / data-prepper

OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
262 stars 202 forks source link

Feature request - End to End ACKs / Queueless Mode #851

Open ryn9 opened 2 years ago

ryn9 commented 2 years ago

Is your feature request related to a problem? Please describe. Not related to a problem - this is a feature request

Describe the solution you'd like As the internal piping for data-prepper is relatively young... please consider supporting End to End ACKs / Queueless Mode for entire pipelines. Please reference: https://github.com/elastic/logstash/issues/8514

Describe alternatives you've considered (Optional) N/A. This would be a core idea for the 'infrastructure' of pipelines in the tool.

Additional context N/A

dlvenable commented 2 years ago

@ryn9 , Thank you for the feature request. I'd like to clarify what you are needing here. Do you need to actually receive an acknowledgement when the sinks receive the data? Or are you looking for more assurances related to guaranteed delivery?

Presently, Data Prepper stores events in an in-memory buffer. So if the Data Prepper instance crashes, then all events currently in Data Prepper are lost. If Data Prepper had either an on-disk buffer or buffer with a distributed queueing system, would this meet your need? The Data Prepper maintainers are looking into more durable buffers.

ryn9 commented 2 years ago

I am looking for assurances related to guaranteed delivery. IE. if there is an http input - it does not give a 200 response until it successfully flushes out the sink.

dlvenable commented 1 year ago

We have been working on adding end-to-end acknowledgements in Data Prepper 2.2. This feature will initially work for the s3 source only. With it, Data Prepper will not notify back to the SQS queue until all the records have been successfully sent to the OpenSearch sink.

ryn9 commented 1 year ago

We have been working on adding end-to-end acknowledgements in Data Prepper 2.2. This feature will initially work for the s3 source only. With it, Data Prepper will not notify back to the SQS queue until all the records have been successfully sent to the OpenSearch sink.

Excited to here it, though I would be most interested in using it for HTTP input.

dlvenable commented 1 year ago

Some of the PRs supporting this feature: