vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.58k stars 1.54k forks source link

Allow `time` field to be set on `splunk_hec` sink #6368

Open jszwedko opened 3 years ago

jszwedko commented 3 years ago

Requested by user: https://github.com/timberio/vector/issues/6185#issuecomment-773566173

The user would like to be able to specify the time Splunk metadata field in the same way as other metadata fields like source and sourcetype can be specified.

For example:

[sinks.splunk_hec]
type = "splunk_hec"
inputs = [...]

time = "{{ alert_timestamp }}"

Currently the splunk_hec source only uses the timestamp key.

Notably, I think this will require some extra parsing since templates return strings, but we need to convert the timestamp to a unix timestamp in that sink. I think optimistically parsing the timestamps with the default timestamp formats supported by the coercer transform could be an approach.

jeromekleinen commented 3 years ago

Another approach could be to add an option like time_key (similar to host_key) where the user should refer to a key that is already coerced into a proper timestamp. The default value for time_key could be timestamp, which also makes it much more clear where the event time is currently coming from.

p.s. small typo in your issue - I think this is about the splunk_hec sink and not the source.

jszwedko commented 3 years ago

Indeed, thanks @jeromekleinen !

And yes, we could support this via timestamp_key as well.

I also opened #6352 as well as I think having both _key fields as well as templatable fields is a bit confusing.