search

opensearch-project / data-prepper

Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
241 stars 177 forks source link

Dynamic key_path_prefix in S3 DLQ #4634

Open graytaylor0 opened 4 weeks ago

graytaylor0 commented 4 weeks ago

Is your feature request related to a problem? Please describe. As a user of the S3 DLQ, I would like to dynamically determine the prefix to send failed documents to with Data Prepper expressions, similar to how the S3 sink path_prefix can be dynamic

Describe the solution you'd like

sink:
  - opensearch:
       s3:
         dlq:
           key_path_prefix: "my-${/key}-prefix/"

Alternative Solution An alternative that would be more useful would be to create a pipeline DLQ that can send to the s3 sink, which supports dynamic path prefix

dlvenable commented 4 weeks ago

If we use the pipeline DLQ (#3857) we could get this for free. And it would be consistent.