Leverage mode "time_series" (TSDB) querying capabilities in integration dashboards

[flash1293]

time_series indices in Elasticsearch (e.g. discussed here: https://github.com/elastic/ingest-dev/issues/1036) offer a bunch of advantages with regard to disk space usage, but there will also be a few new querying capabilities https://github.com/elastic/elasticsearch-adrs/blob/master/analytics/tsdb/tsdb-design.md#aggregation-apis

• Better handling of rate calculation for counter metrics (automatic reset detection)
• time_series agg to split individual series without the need to specify the dimension fields (as these are defined in the mapping)
• Sliding window date histogram (decoupling window size and step width of the date histogram)

The Kibana VisEditors team is currently thinking about how to best support TSDB usage in visualizations. What are common cases of visualizations which are not built / built in a sub-optimal way right now due to limitations of the existing tools?

Right now the following is planned:

• Support for Lens, but not TSVB
• Use Elasticsearch side rate calculation together with time_series and sum_bucket to do a proper rate calculation summed up for all series without user configuration for counter metric fields
• Allow to do time_series + sum_bucket for other metrics (average, "last value")
• Later: Allow configuring a window size for metrics which will transparently use the sliding window date histogram instead of regular date histogram

Note: These new features will only function for time_series data, so they won't show any results for indices with historical data indexed as regular data indices.

[flash1293]

cc @mtojek @ruflin Who could provide some input on this?

[mtojek]

I would ping @ravikesarwani or @akshay-saraswat as they care about the product side of integrations.

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ruflin]

@lalit-satapathy It seems most of the things discussed above have been done / are in use? Should this issue be closed?

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!