Data Prepper function to format a date-time

[dlvenable]

Is your feature request related to a problem? Please describe.

We would like to support Hadoop partitions in the S3 sink. For example, we may wish to write an object to the following key path:

events/year=2023/month=10/day=04

Describe the solution you'd like

Provide a new Data Prepper expression method to format a date-time as desired.

date_time_format(EventKey, FormatString)

You could achieve the above example using a configuration like the following:

path_prefix: "events/year=${date_time_format(eventTime, "YYYY")}/month=${date_time_format(eventTime, "MM")}/day=${date_time_format(eventTime, "dd")}/"

Describe alternatives you've considered (Optional)

One alternative is to update the date processor to create formatted strings. This would mean writing to the Event, but the user doesn't necessarily want the data in the S3 object. It could write to the metadata instead.

path_prefix: "events/year=${getMetadata(\"year\")}/month=${getMetadata(\"month\")}/day=${getMetadata(\"day\")}}/"

Another alternative is to make a new special syntax. But, we already have a functions concept. So I think we should build on this rather than try to make some new syntax.

Additional context

See https://github.com/opensearch-project/data-prepper/issues/3310#issuecomment-1747508402 from #3310.

[dlvenable]

Better yet, support:

path_prefix: "events/${date_time_format(eventTime, "year=YYYY/month=MM/day=DD")}/"

[MrR0807]

Will this feature come to 2.9? If no, what is expected timeline? Because I can wait a month/end of summer. However, if this feature will take more time, I'd rather find alternatives. Thank for reply in advance.

[sharraj]

We will add this feature by month end in the product.

[shenkw1]

spoke with @dlvenable and I will be working on this