search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.57k stars 24.63k forks source link

[ES|QL] High accuracy cardinality aggregation #107231

Open wchaparro opened 5 months ago

wchaparro commented 5 months ago

Description

Use case: financial services users need to use ES (Kibana) for precise high cardinality counts (millions of accounts) for daily reporting needs. While the current cardinality aggregation works fast and supports very large datasets it is not precise. It would be very useful if ES|QL supported a precise cardinality calculation via a new aggregation. In the past we have closed requests for this kind of functionality due to our policy that all features that we implement need to work and scale in a distributed environment. ES|QL opens new opportunities for potentially solving this because We can support long running queries via async_search and we have a way to store intermediate results via ES|QL which means the memory risk is much lower.

See https://github.com/elastic/kibana/issues/179934, related: https://github.com/elastic/elasticsearch/issues/60149

elasticsearchmachine commented 5 months ago

Pinging @elastic/es-analytical-engine (Team:Analytics)