opensearch-project / data-prepper

OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
267 stars 206 forks source link

Documentation on how to configure pipelines with disk based buffering #2402

Open talkraghu opened 1 year ago

talkraghu commented 1 year ago

Describe the bug https://opensearch.org/docs/2.4/data-prepper/pipelines/configuration/buffers/buffers/

Buffers store data as it passes through the pipeline. If you implement a custom buffer, it can be memory based, which provides better performance, or disk based, which is larger in size.

The above said documentation states that "disk based" buffering can be configured instead of a memory based "bounded_blocking" one. But I don't see the parameter names or ways to configure it

Can someone pls give an example where pipeline parts like "entry-pipeline", "raw-pipeline" and "service-pipeline" are configured with disk based buffering.

dlvenable commented 1 year ago

@talkraghu , There is currently no disk-based buffering. I believe we need to update that documentation. Thanks for pointing it out.