search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.67k stars 8.23k forks source link

[ES|QL] Should calculate true value for total number of docs for given date range #195787

Open sophiec20 opened 1 month ago

sophiec20 commented 1 month ago

Seen in 8.16 and prior.

The displayed value for total number of documents should not be limited to 1000.

When writing an ES|QL query, a | LIMIT 1000 is added behind-the-scenes. This is a UI-applied guardrail to protect against excessive query load, in place until such time as paging is available and/or optimizations for timestamp ordering.

The LIMIT 1000 does not apply to the time chart because there is an expectation to see the full event rate for the given time range - this mirrors existing KQL functionality.

The LIMIT 1000 applies to the table view - this mirrors existing KQL functionality and avoids too much data being brought back to the UI. This is ok.

However the LIMIT 1000 applies to the doc count (number of docs shown in the tab heading) which seems to be incorrect. Whilst this is the number of docs in the table, it should really show the full doc count for the given time range. This would mirror existing KQL functionality and would show the expected number of docs.

Image

elasticmachine commented 1 month ago

Pinging @elastic/kibana-esql (Team:ESQL)

elasticmachine commented 1 month ago

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

stratoula commented 1 month ago

Related to this https://github.com/elastic/kibana/issues/166219

kertal commented 1 month ago

We were discussing this, and identified this issue as being a part of https://github.com/elastic/kibana/issues/196444 with the aim to align DataView and ES|QL behavior