Buffers store data as it passes through the pipeline. If you implement a custom buffer, it can be memory based, which provides better performance, or disk based, which is larger in size.
The above said documentation states that "disk based" buffering can be configured instead of a memory based "bounded_blocking" one. But I don't see the parameter names or ways to configure it
Can someone pls give an example where pipeline parts like "entry-pipeline", "raw-pipeline" and "service-pipeline" are configured with disk based buffering.
Describe the bug https://opensearch.org/docs/2.4/data-prepper/pipelines/configuration/buffers/buffers/
Buffers store data as it passes through the pipeline. If you implement a custom buffer, it can be memory based, which provides better performance, or disk based, which is larger in size.
The above said documentation states that "disk based" buffering can be configured instead of a memory based "bounded_blocking" one. But I don't see the parameter names or ways to configure it
Can someone pls give an example where pipeline parts like "entry-pipeline", "raw-pipeline" and "service-pipeline" are configured with disk based buffering.