elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
838 stars 24.8k forks source link

Add support for aggregate_metric_double field in es|ql #110649

Open martijnvg opened 3 months ago

martijnvg commented 3 months ago

All downsampled indices will contain fields of type aggregate_metric_double, which is the result of downsampling numeric fields with time_series_metric attribute set to gauge. Supporting aggregate_metric_double is required to supported tsdb in es|ql.

ES|QL has a much more advanced type system than aggregations. Aggregations almost always return numeric responses as double, whereas es|ql returns numeric fields in the type that they are mapped. This means aggregate_metric_double field type isn't sufficient as it would only allow to query rational gauges that have downsampled and not whole number gauges that have been downsampled.

The aggregate_metric_double field type was sufficient for aggregations, but with es|ql's more sophisticated type system it isn't enough. Therefor we should introduce aggregate_metric_long, which is similar to aggregate_metric_double but for gauges that are whole numbers.

Additionally es|ql needs to understand to the gauge fields are typed differently depending on whether a backing index has been downsampled. For example the first few indices a gauge may be mapped as double, but in the last backing index it is mapped as aggregate_metric_double. We should allow queries to hit all these indices with failing with validation errors.

Tasks:

elasticsearchmachine commented 3 months ago

Pinging @elastic/es-storage-engine (Team:StorageEngine)