search

elastic / ecs

Elastic Common Schema
https://www.elastic.co/what-is/ecs
Apache License 2.0
997 stars 413 forks source link

Support for nanosecond timestamps (date_nanos) #1065

Open segevfiner opened 3 years ago

segevfiner commented 3 years ago

Description of the issue: Elasticsearch since v7 has added support for a date_nanos type, supporting timestamps with nanosecond resolution, but the ECS @timestamp is a date supporting only millisecond resolution.

It would be nice if ECS had official fields for storing nanosecond resolution timestamps that would also be properly supported by beats, beats modules and the configurations they preload into Kibana.

How this should interact with the existing required @timestamp field is something to be considered.

Any additional context or examples: https://www.elastic.co/guide/en/ecs/current/ecs-base.html https://www.elastic.co/guide/en/elasticsearch/reference/current/date_nanos.html

segevfiner commented 3 years ago

This is a feature request, not a bug. Auto-labeling 🤷‍♂️

webmat commented 3 years ago

event.duration is nanoseconds, I could have sworn @timestamp was also already nanoseconds precision 🤦

Thanks for pointing this out. We'll discuss this, see what the possibilities are in changing this.

For now of course, you're free to add a custom field with nanosecond precision for your use case. The guidance on naming custom fields is here.

webmat commented 3 years ago

At first glance it looks like date and date_nanos are actually compatible. I'm not yet convinced they can be used 100% interchangeably, for example date_nanos seems to top out around year 2262 because of different usage of the integer range.

We'll do some more digging to determine next steps. But you may be able to swap out date for date_nanos for the @timestamp field (on a test setup), and things should work. That's definitely something I will experiment with soon 🔥 😉 .