search

elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
68.56k stars 24.35k forks source link

Multiple timestamp support in TSDB #95389

Open wchaparro opened 1 year ago

wchaparro commented 1 year ago

Description

Multiple timestamps may be common in situations where devices or systems that may be down or unavailable for periods of time, and at a later time is brought back up and resumes processing. These situations are quite common in IOT and monitoring systems, and so we should find ways to optimize / handle these timestamps... as a timestamp itself takes significant space. A recent example of this was brought to us where the @timestamp represents the actual read time on the device whereas @received_time is when the data was processed. @giladgal @martijnvg

elasticsearchmachine commented 1 year ago

Pinging @elastic/es-analytics-geo (Team:Analytics)

martijnvg commented 1 year ago

I think in many cases there is always a relation between the two timestamps. If that is the case, then maybe we can store the delta or offset between @timestamp and the additional timestamp.

Also in certain cases multiple different metrics with the same timestamp and dimensions stored as different documents. In those cases the timestamp offset or delta can maybe also act as an additional component in the _id that tsdb generates. At query time these different metric documents do end up in the same time series.

elasticsearchmachine commented 3 months ago

Pinging @elastic/es-storage-engine (Team:StorageEngine)