opensearch-project / data-prepper

OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
262 stars 200 forks source link

[BUG] In OpenSearch pipelines, ${pipelineName} variable does not get replaced in s3 key_path_prefix #3609

Open heather-slatford opened 1 year ago

heather-slatford commented 1 year ago

Describe the bug

The ${pipelineName} variable does not get replaced in this field:

  sink:
    - opensearch:
        ...
        dlq:
          s3:
            bucket: "our_dlq_bucket"
            key_path_prefix: "${pipelineName}/logs/dlq"

To Reproduce Steps to reproduce the behavior: Create a pipeline with a DLQ for the sink and use the ${pipelineName} variable to construct an object key prefix.

Object keys will contain "${pipelineName}" instead of the pipeline name.

Expected behavior Object key should contain the actual pipeline name

Screenshots n/a

Environment (please complete the following information): n/a

Additional context This is trivial to hard-code the pipeline name in each pipeline definition, but most users will expect this to work like it does for the ingestion path: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/creating-pipeline.html#pipeline-path

dlvenable commented 11 months ago

Right now the ${pipelineName} variable is not globally available. We should make this available for variable expansion anywhere.