search

opensearch-project / data-prepper

Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
256 stars 188 forks source link

Support dynamic indices with more granular time-based partitions #3558

Open graytaylor0 opened 10 months ago

graytaylor0 commented 10 months ago

Is your feature request related to a problem? Please describe. As a user of the OpenSearch sink, I would like to roll over my indices with custom time durations, rather than just those provided by the JavaDateTime format (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html). For example, I would like to either roll over indices every 2 days, or every 6 hours, instead of being forced to use granularity of 1 day or 1 hour only.

Describe the solution you'd like The ability to create dynamic indices and rollover with specific time patterns, but without the restriction of using a multiple of 1 for hours and days.

Describe alternatives you've considered (Optional) A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

dlvenable commented 10 months ago

I think we could do something similar to the Data Prepper duration concept. Let a user specify PT2D or 2d.