jvalente-salemstate
commented
9 months ago After opening this I see #15701 was also opened--it didn't come up in a search as it's about the @timestamp field but provides another example of where setting this field could be helpful, as well as an example of handling overwrites should logstash's ingestion be the desired timestamp or creation time.
They're slightly different issues but overlap in some use cases.
The description of the field in the ECS documentation is
From the couple of inputs I've checked there isn't an option to do this. The next best option is adding it within the first filter of the pipeline. However this isn't exactly what is intended for the field (though a few ms difference is trivial) and could, at least conceptually, obfuscate any delays between the input and the next filter adding the field.
What I am suggesting is adding an option to each input plugin that will populate
[event][created]with the current time at ingestion. This aligns with the intended use of the field and improves its use for pipeline monitoring.This probably should be an optional field that defaults to false to avoid issues for pipelines that have already added the field or do not want to use it. Possibly, the field name itself could also differ (or be a setting) depending on the input plugin/pipeline's ecs_compatibility settings.
There may also need to consideration of handling input that may already have it included (beats, agent, another logstash node) and whether to skip setting it or overwrite it. In terms of ecs it's probably better to skip that unless the pipeline's specific use treats this ingestion as the creation. Though, largely, I am guessing any of those cases just would not set the option if it was already set in the pipeline.