search

opensearch-project / data-prepper

OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
266 stars 205 forks source link

[BUG] Invalid time pattern placement in dynamic index pattern shuts down pipeline #3386

Open engechas opened 1 year ago

engechas commented 1 year ago

Describe the bug DataPrepper does not allow time patterns to be in dynamic index patterns unless they are the suffix of the pattern: https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/AbstractIndexManager.java#L127

This causes the pipeline to shutdown at runtime when data is sent to a sink who's index pattern violates this condition.

ERROR org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor - Pipeline [***] process worker encountered a fatal exception, cannot proceed further
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Time pattern can only be a suffix of an index

To Reproduce Create a pipeline with this index pattern in an opensearch sink:

prefix-%{yyyy.MM.dd}-suffix

Expected behavior It's unclear why DataPrepper does not support non-suffix time patterns. If this is not a hard restriction then I would expect this validation to be removed. If it is a hard restriction then I would expect this to be validated during initialization rather than throwing an exception and shutting down the pipeline at runtime.

dlvenable commented 1 year ago

I believe that this is a suffix because that will work with our index and ISM patterns. I think the simplest solution is to solve the bug by validating on start-up.

We could consider an enhancement to support non-suffix in the future.