search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.41k stars 1.51k forks source link

Remove strftime specifiers from the templating syntax #6254

Open binarylogic opened 3 years ago

binarylogic commented 3 years ago

The current support for strftime specifiers in the templating syntax was a quick hack to allow for date-based partitioning:

[sinks.backup]
  type = "aws_s3"
  bucket = "all_application_logs"
  key_prefix = "application_id={{ application_id }}/date=%F/"

The obvious limitation of this is the assumption that the timestamp field should be used to derive their formatted dates. What if a user wants to use another field?

Given that we decided against supporting the Remap syntax in #3836, we should do 1 of 2 things:

  1. Remove support for the strftime specifiers and require users to format dates in a remap transform.
  2. Reopen the discussion of supporting Remap functions in the templating syntax.

I lean towards 1 until we see more reasons to do 2.

jszwedko commented 3 years ago

I think we need to address the metadata vs. event field separation before moving towards solution 1 here as otherwise it'll end up requiring users to write a "dummy key" in the remap transform that they probably will not want to actually appear in the event when it is written to S3 (in this example)